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PREFACE 


This  Handbook  is  about  source  selection  cost  evaluation  for 
competitive,  cost  reimbursable  contracts  with  particular 
emphasis  on  assuring  more  realistic  contractor  cost  proposals. 
It  is  intended  as  a  practical  guide  and  reference  for  program 
management  personnel  and  source  selection  personnel.  It 
presents  available  techniques  and  procedures  and  how  to  apply 
them.  The  techniques  are  both  qualitative  and  quantitative  in 
nature . 

Some  readers  may  prefer  to  read  only  the  summary  informa¬ 
tion  to  obtain  an  overview  on  the  subject,  whereas  others  may 
want  to  read  in  detail  so  as  to  be  able  to  apply  the  techniques. 

This  Handbook  was  developed  and  authored  by  Science  Appli¬ 
cations  International  Corporation,  Arlington,  Virginia,  under 
contract  number  N00014-83-C-0123  directed  by  the  Navy  Office  of 
Acquisition  Research'."  ""TITe  principal  author  is  Donald  Trapp  who 
was  supported  by  James  Rydzewski  and  David  Jordan. 

Specie  1  thanks  are  due  to  the  many  Defense  Department 
personnel  who  provided  information  for  this  Handbook  and  who 
reviewed  the  work  at  various  stages.  In  particular,  this  Hand¬ 
book  has  benefited  greatly  from  the  assistance  and  advise  of 
Irwin  Neveleff  of  the  Naval  Electronic  Systems  Command  who  has 
applied  cost  realism  as  a  cost  proposal  evaluation  technique  for 
the  past  ten  years. 

The  Navy  Office  of  Acquisition  Research  is  the  controlling 
agency  for  this  Handbook.  Comments  and  recommendations  relat¬ 
ing  to  the  Handbook  are  solicited.  Based  on  your  responses, 
revisions  to  the  Handbook  may  be  published. 


Navy  Office  of  Acquisition  Research 
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EXECUTIVE  SUMMARY 


This  Cost  Realism  Handbook  provides  the  Defense  acquisition 
community  with  a  methodology  for  evaluating  contractor  cost 
proposals,  with  particular  emphasis  on  asuring  more  realistic 
cost  proposals.  This  should  be  of  particular  interest  to  source 
selection  cost  evaluation  teams,  as  well  as  to  contracting 
officers  and  program  managers.  The  methodology  is  applicable 
mainly  to  competitive,  cost  reimbursable  contracts.  The  Hand¬ 
book  explains  the  various  steps  of  the  methodology  in  detail, 
provides  an  illustrative  example,  and  describes  computer  models 
which  may  be  useful  in  establishing  scoring  rules  and  evaluation 
factor  weights. 

The  methodology  includes  four  interrelated  steps: 

(1)  Evaluation  Factors:  Determination  of  cost  eval¬ 

uation  factors; 

(2)  Solicitation:  Preparation  of  instructions  to  be 
included  in  the  solicitation  concerning  the  cost 
evaluation  factors; 

(3)  Government  Estimate:  Preparation  of  a  Government 
estimated  cost  for  each  contractor;  and 

(4)  Evaluation:  Rating  or  scoring  each  contractor  for 
Government  estimated  cost  and  cost  realism. 

The  methodology  involves  the  best  techniques  and  procedures 
currently  being  used  in  source  selection  cost  evaluation.  In 
addition  this  Handbook  introduces  several  improvements  to  the 
current  techniques,  especially  in  the  area  of  numerical  scoring. 

It  is  believed  that  application  of  this  methodology  will 
encourage  contractors  to  make  more  realistic  offers  since  they 
will  be  on  notice,  though  the  solicitation,  that  cost  realism 
will  be  examined  and  could  be  a  deciding  factor  in  the  award. 
The  contractors  will  know  that  the  evaluation  procedures  will 
reduce  the  likelihood  of  awards  to  low  offers  on  cost  reimburs- 
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able  contracts  unless  realistically  low,  if  cost  is  the  deciding 
factor . 

There  are  many  factors  involved  in  a  source  selection  and 
each  procurement  is  unique.  Therefore,  the  cost  evaluation 
methodology  must  be  practical  and  flexible.  Also,  the  results 
of  applying  the  methodology  must  be  logical.  The  methodology 
described  in  this  Handbook  will  accomplish  these  objectives  if 
it  is  used  as  a  general  guide  and  applied  as  appropriate  to  each 
procurement . 

For  a  quick  overview  of  the  Handbook  read  the  INTRODUC¬ 
TION  (Section  I),  the  METHODOLOGY  SUMMARY  (Section  IV)  and 
the  ILLUSTRATIVE  EXAMPLE  (Section  X). 


For  a  more  detailed  understanding  of  the  methodology,  read 
the  separate  sections  on  each  of  the  four  steps.  Cost  analysts 
who  are  involved  or  expect  to  be  involved  in  a  source  selection 
should  study  the  following  sections  in  detail:  EVALUATION 
(Section  VIII)  and  SENSITIVITY  AND  TRADEOFF  ANALYSIS  (Section 
IX). 
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SECTION  I 
INTRODUCTION 


Unrealism  in  Defense  contractors'  cost  proposals,  especial¬ 
ly  for  RDT&E  programs,  often  contributes  to  cost  growth  and 
related  problems.  Cost  growth  may  require  drastic  changes  in 
system  design  and  program  scope.  It  also  may  lead  to  schedule 
slippages  and  program  cancellations.  And,  unrealistically  high 
offers  may  lead  to  the  Government  paying  too  much.  The  Defense 
Department  is  therefore  concerned  with  achieving  greater  cost 
realism . 

This  Cost  Realism  Handbook  provides  the  Defense  acquisition 
community  with  a  methodo 1 ogy--that  is  techniques  and  procedures 
--for  evaluating  contractor  cost  proposals,  with  particular  em¬ 
phasis  on  asuring  more  realistic  cost  proposals.  This  should  be 
of  particular  interest  to  source  selection  cost  evaluation 
teams,  as  well  as  to  contracting  officers  and  program  managers. 
The  methodology  is  applicable  mainly  to  competitive,  cost  reim¬ 
bursable  contracts.  The  Handbook  explains  the  various  steps  of 
the  methodology  in  detail,  provides  an  illustrative  example,  and 
describes  computer  models  which  may  be  useful  in  establishing 
scoring  rules  and  evaluation  factor  weights. 

A.  BACKGROUND 

How  realistic  are  contractor  cost  proposals?  This  is  not 
easy  to  determine.  The  complexity  and  risks  inherent  in  Defense 
systems  make  contractor  cost  estimation,  not  to  mention  Govern¬ 
ment  review  and  evaluation  of  the  costs,  difficult.  Some 
optimism  may  be  warranted  (e.g.,  in  engineering  hours  required, 
in  number  of  test  articles,  and  in  number  of  tests),  but  when 
does  optimism  become  unrealism?  In  some  cases,  the  contractor 
may  purposely  propose  an  unrealistically  low  cost  in  order  to 
"buy-in"  and  thereby  improve  its  business  base  and  cash  flow. 

Data  have  been  collected  on  proposed  costs  and  Government 
estimates  for  eight  programs  from  the  Naval  Electronic  Systems 
Command  (NAVELEX)  which  provide  some  insight  into  the  degree  of 
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realism  found  in  contractor  cost  proposals  for  electronic 
systems.  In  developing  its  estimates,  NAVELEX  evaluated  the 
contractor  cost  proposals  in  detail,  required  the  contractors  to 
justify  their  data  and  estimating  rationale,  developed  their  own 
estimates  based  almost  entirely  on  the  contractor's  data,  and 
compared  these  estimates  with  the  cost  proposals.  The  data 
showed  that  one  out  of  three  original  offers  was  more  than  20 
percent  low,  and  one  out  of  six  best-and-final  offers  was  more 
than  20  percent  low.  Offers  more  than  20  percent  below  the 
NAVELEX  estimate  certainly  raise  a  question  as  to  cost  realism. 
Interestingly,  less  than  one-quarter  of  the  original  offers  but 
nearly  half  of  the  best-and-final  offers  were  within  3  percent 
of  the  NAVELEX  estimates,  apparently  as  a  result  of  the  detailed 
Government  cost  evaluation  and  the  fact  that  cost  realism  was  an 
evaluation  criterion. 

Of  course,  the  Government  must  also  guard  against  unrealis¬ 
tically  high  cost  estimates.  This  situation  will  most  probably 
occur  if  there  are  few  technically  qualified  contractors  and  if 
a  contractor  already  has  an  especially  good  business  base.  High 
cost  estimates  do  minimize  the  likelihood  of  an  overrun  but  the 
Government  could  probably  have  bought  the  system  for  less. 

How  do  we  achieve  greater  cost  realism?  This  poses  another 
difficult  question,  especially  since  there  are  often  disincen¬ 
tives  to  realistic  cost  proposals  in  source  selections  for  cost- 
reimbursable  contracts.  These  disincentives  occur  because  cost 
is  generally  a  secondary  consideration,  especially  for  RDT&E 
contracts.  Additionally,  the  Government  often  tends  to  award  to 
the  contractor  with  the  lowest  cost  proposal,  if  technical  and 
other  factors  are  not  too  different;  or  at  least,  the  contrac¬ 
tors  may  perceive  that  the  Government  has  such  a  bias. 

The  Defense  Department  needs  to  continue  to  develop  and 
refine  acquisition  policies  and  practices  which  encourage  and 
achieve  more  cost  realism.  In  particular,  source  selection 
procedures  should  be  improved  so  as  to  do  more  to: 
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Encourage  realistic  bids, 

Discourage  awards  to  the  low  bidder  on  RDT&E 
contracts  unless  realistically  low,  if  cost  is  the 
deciding  factor,  and 

Discourage  success  orientation  bias. 

It  is  primarily  through  the  source  selection  cost  evalua¬ 
tion  process  that  the  Government  can  direct  efforts  to  give 
greater  assurance  that  cost  proposals  are  reasonable,  defensible 
and  realistic.*  In  a  general  sense,  then,  cost  realism  may  be 
defined  as  that  part  of  the  source  selection  process  directed  at 
giving  greater  assurance  of  the  reasonableness  and  d e f ens i b i 1 i t y 
of  contractors’  cost  proposals, 

B.  METHODOLOGY 

A  source  selection  cost  evaluation  methodology  is  described 
in  this  Handbook  which  should  assure  a  sound  cost  evaluation  and 
should  encourage  contractors  to  make  more  realistic  offers. 
This  methodology  includes  four  interrelated  steps: 

(1)  Evaluation  Factors:  Determination  of  cost  eval¬ 

uation  factors; 

(2)  Solicitation:  Preparation  of  instructions  to  be 

included  in  the  solicitation  concerning  the  cost 
evaluation  factors; 

(3)  Government  Estimate:  Preparation  of  a  Government 
estimated  cost  for  each  contractor;  and 

(4)  Evaluation:  Rating  or  scoring  each  contractor  for 
Government  estimated  cost  and  cost  realism. 

The  methodology  involves  the  best  techniques  and  procedures 
currently  being  used  in  source  selection  cost  evaluation, 

*  Acquisition  policies  and  practices  other  than  those 
associated  with  source  selection  cost  evaluation  may  also  help 
in  assuring  more  realistic  cost  proposals  (e.g.,  certain 
incentive  type  contracts  and  the  application  of  risk  analysis  in 
the  proposal  preparation).  These  other  policies  and  practices 
are  discussed  briefly  in  Section  XI. 
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especial’/  those  of  NAVELEX.  In  addition,  this  Handbook  intro¬ 
duces  several  improvements  to  the  current  techniques. 


NOTE:  The 

terms  "Government 

estimated 

cost" 

and 

"Government 

estimate" 

are  used  in  this 

Handbook 

for 

the 

Government 

estimate  of 

the  likely  cost 

for  each  contractor. 

A  definition  of  cost  realism  was  given  above.  The  method¬ 
ology  suggests  a  more  specific  definition  of  cost  realism  as  an 
evaluation  criterion  stated  in  the  solicitation  which  compares 
the  contractor's  proposed  cost  with  a  detailed  Government  esti¬ 
mate  for  each  contractor  and  then  rates  or  scores  the  degree  of 
realism. 

It  is  believed  that  application  of  this  methodology  will 
encourage  contractors  to  make  more  realistic  offers  since  they 
will  be  on  notice,  though  the  solicitation,  that  cost  realism 
will  be  examined  and  could  be  a  deciding  factor  in  the  award. 
The  contractors  will  know  that  the  evaluation  procedures  will 
reduce  the  likelihood  of  awards  to  low  offers  on  cost  reimburs¬ 
able  contracts  unless  realistically  low,  if  cost  is  the  deciding 
factor. 

There  are  many  factors  involved  in  a  source  selection  and 
each  procurement  is  unique.  Therefore,  the  cost  evaluation 
methodology  must  be  practical  and  flexible.  Also,  the  results 
of  applying  the  methodology  must  be  logical.  The  methodology 
described  in  this  Handbook  will  accomplish  these  objectives  if 
it  is  used  as  a  general  guide  and  applied  appropriately  to  each 
procurement . 

The  next  section  of  the  Handbook  provides  background  on  the 
source  selection  process.  The  third  section  discusses  cost 
realism  and  cost  growth.  Next,  the  methodology  is  summarized. 
This  is  followed  by  four  sections  which  discuss  each  of  the  main 
components  of  the  methododo logy:  evaluation  factors,  solici¬ 
tation,  Government  estimate,  and  evaluation.  Then,  sensitivity 
and  tradeoff  analyses  are  discussed.  An  illustrative  example  is 
provided  in  Section  X.  Several  methods  for  assuring  greater 


cost  realism,  other  than  those  involving  source  selection  cost 
evaluation,  are  then  discussed.  Conclusions  are  presented  in  a 
final  section.  There  are  five  appendicies  which  provide  defini¬ 
tions  of  key  terms,  a  listing  of  acronyms  and  symbols,  policy 
document  excerpts,  a  cost-benefit  analysis,  and  a  complete  de¬ 
scription  of  the  cost  realism  computer  models  including  sample 
outputs . 
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SECTION  II 
SOURCE  SELECTION 


This  section  provides  some  background  on  Government 
procurement  policies  and  discusses  the  source  selection  process. 
Then,  cost  considerations  in  source  selection  are  discussed. 

A.  BACKGROUND 

A  procurement  process  has  evolved  in  the  Department  of 
Defense  (DOD)  for  acquiring  necessary  supplies  and  services  of 
the  desired  quality  in  a  timely  manner,  and  at  fair  and  reason¬ 
able  prices.  This  procurement  process  involves  certain  proce¬ 
dures  for  translating  requirements  into  contracts.  The  overall 
objective  of  this  process  is  to  obtain  the  best  value,  given 
cost,  technical  and  other  considerations.  There  are  two  major 
methods  of  procurement — "formal  advertising”  and  "negotiation". 

Formal  advertising  is  the  preferred  method  where  competi¬ 
tion  is  present,  the  supplies  or  services  can  be  adequately  and 
precisely  specified,  and  the  selection  can  be  made  solely  on 
price.  The  Government  issues  an  invitation  for  bids  (IFB), 
contractors  respond  with  price  bids,  and  the  Government  awards 
the  contract  to  the  lowest  bidder.  Bid  prices  do  not  have  to  be 
substantiated  by  the  bidders.  The  Government  does  not  analyze 
the  costs,  and  the  costs  are  not  subject  to  negotiation. 

Procurement  by  negotiation  is  used  when  it  is  impractical 
to  secure  competion  by  formal  advertising,  when  the  products 
being  procured  are  new  and  complex,  and  under  certain  other 
circumstances  as  specified  in  the  Federal  Acquisition  Regula¬ 
tions  (FAR).  It  is  especially  appropriate  for  RDT&E  programs. 
In  procurement  by  negotiation,  the  Government  issues  a  solicita¬ 
tion,  either  a  request  for  proposals  (RFP)  or  a  request  for 
quotations  (RFQ);  contractors*  respond  with  technical  and  cost 


*  A  contractor  who  submits  a  proposal  or  offer  to  the 
Government  in  response  to  a  Government  solicitation  is  often 
referred  to  as  an  "offeror".  The  terms  contractor  and  offeror 
are  often  used  interchangeably  in  this  Handbook. 
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proposals  (often  very  detailed);  and  discussions  and  negotia¬ 
tions  (often  extensive)  are  conducted  between  the  Government  and 
the  contractors  involving  all  aspects  of  the  proposals  (e.g., 
technical,  cost  and  schedule).  The  Government  then  awards  the 
contract  primarily  on  the  basis  of  a  contractor's  ability  to 
perform  the  work,  as  demonstrated  by  its  proposal.  The  Govern¬ 
ment  requires  that  the  contractor  cost  proposals  be  substan¬ 
tiated  with  detailed  data  and  a  discussion  of  the  estimating 
methodology.  The  Government  analyzes  these  costs  and  negotiates 
them  with  the  contractors. 

B.  SOURCE  SELECTION  PROCESS 

The  source  selection  process  applies  to  negotiated  procure¬ 
ments  where  contract  award  will  be  based  on  factors  in  addition 
to  price.  In  order  to  arrive  at  a  best  possible  award  decision, 
a  common  source  selection  organization  and  set  of  procedures 
have  evolved  among  the  Services.  These  policies  and  procedures 
have  been  developed  primarily  for  major  procurements,  as  defined 
by  DOD  Directive  5000.1,  Major  Systems  Acquisitions  (e.g.,  RDT&E 
exceeding  $200  million  in  FY80  dollars).  However,  less-than- 
raajor  procurements  often  follow  many  of  the  same  policies  and 
procedures.  It  should  be  noted  that  there  is  a  certain  amount 
of  flexibility  in  the  source  selection  organization  and  proce¬ 
dures  (e.g.,  in  the  size  and  composition  of  the  source  selection 
team  and  in  the  evaluation  methods  used). 

The  source  selection  organization  typically  comprises  the 
elements  shown  below: 

Source  Selection  Authority  (SSA) 

Source  Selection  Advisory  Council  (SSAC) 

Source  Selection  Evaluation  Board  (SSEB) 

Procurement  Contracting  Officer  (PC0) 

Pro ject/Program  Manager  (PM) 

The  source  selection  organization  executes  a  series  of  steps 
that  culminates  in  an  award  decision.  These  steps  are  listed 
below : 
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Source  Selection  Plan  Preparation  and  Appxoval 

Solicitation  Preparation  and  Approval 

Proposal  Evaluation 

Competitive  Range  Determinations 

Discussions/Negotiations 

Best  and  Final  Proposal  Evaluation 

Source  Selection  and  Award 


C.  COST  CONSIDERATIONS  IN  SOURCE  SELECTION 

Prior  to  beginning  source  selection,  cost  is  important  for 
planning,  programming  and  budgeting.  During  this  time,  there  is 
limited  project  definition,  but  a  cost  estimate  for  a  notional 
system  is  necessary  for  planning  and  early  budgeting.  And,  of 
course,  cost  is  an  important  consideration  throughout  the  source 
selection  process,  from  preparation  of  the  Source  Selection  Plan 
to  award.  Generally,  there  is  an  extensive  effort  to  assure 
that  the  best  value  is  identified  and  that  the  costs  are  reason¬ 
able  and  realistic,  so  as  to  reduce  the  possibility  of  either 
large  overruns  or  accepting  too  high  an  offer.  However,  cost  is 
only  one  evaluation  factor  in  source  selections  and  is  usually 
not  the  driving  factor  for  RDT&E  programs. 

Some  of  the  steps  in  the  procurement  process  leading  to  the 
selection  of  a  source  where  cost  is  important  are  discussed 
below : 

Early  Project  Definition.  Preceding  a  decision  to 
procure  a  system  is  the  need  to  provide  technical 
definition  of  the  item  and  from  that  to  provide 
preliminary  cost  estimates  for  use  in  planning  and 
budgeting.  These  cost  estimates  are  typically  based 
on  gross  information  and  top-level  cost  estimating 
techniques  (e.g.,  parametric)  and  require  later 
refinement  as  better  project  definition  occurs.  These 
early  cost  estimates  represent  the  Government  estimate 
and  best  judgment  as  to  probable  cost.  These  early 
estimates  may  affect  the  Government  assessment  of  the 
reasonableness  of  offerors’  costs.  They  may  also 
affect  the  offerors'  cost  proposals  since  the  budget 
may  be  known  to  them. 

Source  Selection  Plan  Preparation.  In  preparing  the 
Source  Selection  Plan,  the  cost  evaluation  criteria, 
the  weights  to  be  given  the  various  cost  criteria,  and 
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the  method  of  ranking  or  scoring  proposals  must  be 
determined . 

Solicitation  Preparation.  The  solicitation  formalizes 
the  groundrules  for  the  offerors'  cost  submission. 

While  there  is  a  significant  body  of  existing  policy 
which  gives  the  Government  leverage  in  pursuing  cost 
questions  (e.g.,  data,  traceability,  estimation 
methodology),  a  well  thought  through  and  prepared 
solicitation  can  facilitate  the  Government  obtaining 
what  it  needs  to  prevent  either  cost  overrun  from  a 
buy-in  or  paying  too  much.  Also,  the  solicitation  can 
create  an  atmosphere  regarding  the  importance  of  costs 
that  can  influence  the  offerors'  attitudes  towards  the 
reasonableness  of  costs  in  the  proposal.  Specification 
of  a  project  work  breakdown  structure  (WBS)  for  por¬ 
traying  cost  estimates,  and  specifying  that  cost 
estimates  must  be  auditable  and  defendable  can  help 
set  a  proper  atmosphere. 

Proposal  Price/Cost  Analysis*  Upon  receipt  of 
offerors'  cost  proposals  (original,  revisions,  or 
best-and-f ina 1 )  ,  analysis  is  undertaken  to  validate 
the  reasonableness,  accuracy,  and  currency  of  the 
submissions.  The  cost  proposal  review  is  integrated 
with  the  technical  review  to  assure  the  offerors'  cost 
estimates  reflect  the  offerors'  technical  plans.**  A 
variety  of  analysis  techniques  are  called  upon  in  the 
source  selection  process.  The  Federal  Acquisition 
Regulations  (FAR)  provides  for  aggregate  analyses 
(i.e.,  price  analysis)  or  more  detailed  cost  element 
review  (i.e., cost  analysis).  Other  cost  review  tech¬ 
niques  are  selectively  used  such  as  "Will  Cost"  and 
"Should  Cost"  reviews  (which  are  a  form  of  cost  analy¬ 
sis)  in  order  to  test  the  d e f e n s i b i 1  i  t y  of  the 
offerors'  price/cost  proposals. 

Proposal  Evaluation.  There  is  a  need  to  arrive  at  a 
judgment  of  the  quality  and  reasonableness  of  the 
offerors'  cost  proposals.  And,  some  method  is  neces¬ 
sary  to  communicate  this  judgment  to  the  SSA.  This  is 
generally  done  by  ranking  the  offerors  through  the 
assignment  of  numerical  values  or  adjective  assess¬ 
ments  (e.g.,  good,  fair).  Whether  or  not  such  a 
ranking  is  done,  narrative  descriptions  of  the 
analysis  results  are  usually  presented  to  the  SSA. 

Source  Selection  and  Award.  Based  on  assessments  of 
technical,  cost,  management  and  other  factors,  the  SSA 

*  See  Appendix  A  for  definitions  of  price  and  cost 
analysis . 

**  Technical  proposal  evaluators  are  restricted  from  seeing 
the  overall  cost  proposals. 


SECTION  III 
COST  REALISM 


This  section  first  suggests  definitions  for  the  term  "cost 
realism".  Then,  the  problem  of  cost  growth  is  discussed,  and 
the  degree  of  realism  observed  in  cost  proposals  is  analyzed. 
Also,  the  extent  to  which  cost  realism  is  reflected  in  Govern¬ 
ment  policy  is  discussed. 

A.  COST  REALISM  DEFINED 

It  is  in  the  source  selection  steps  discussed  in  the  pre¬ 
vious  Section  that  the  Government  can  direct  efforts  to  give 
greater  assurance  that  cost  proposals  are  reasonable,  defensible 
and  realistic.  In  a  general  sense,  then,  cost  realism  may  be 
defined  as  that  part  of  the  source  selection  process  directed  at 
giving  greater  assurance  of  the  reasonableness  and  defensibilty 
of  contractors'  cost  proposals.  Synonyms  for  realism  are 
reasonableness,  def ensibi 1 ity ,  credibility,  and  validity. 

The  term  cost  realism  has  also  developed  a  more  specific 
meaning  in  organizations  like  NAVELEX  and  NAVAIR.  They  have 
included  cost  realism  as  an  explicit  evaluation  criterion  in  the 
solicitation,  have  prepared  detailed  Government  cost  estimates 
which  are  compared  to  the  contractors'  proposals,  and  have 
evaluated  and  rated  cost  realism  along  with  estimated  cost, 
technical  and  other  factors.  Thus,  a  more  specific  definition 
of  cost  realism  is  an  evaluation  criterion  stated  in  the  solici¬ 
tation  which  compares  the  contractor's  proposed  cost  with  a 
detailed  Government  estimate  for  each  contractor  and  then  rates 
or  scores  the  degree  of  realism. 

As  used  throughout  this  Handbook,  the  term  cost  realism 
will  generally  be  used  only  in  the  second,  more  specific  sense 
where  cost  realism  has  been  made  an  explicit  evaluation 
criterion  in  the  source  selection  process. 
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B.  COST  GROWTH 

As  mentioned  in  the  Introduction,  un realism  in  Defense 
contractors'  cost  proposals,  especially  for  RDT&E  programs, 
often  contributes  to  cost  growth  and  related  problems. 

In  the  past,  many  Defense  acquisition  programs  have  shown 
tremendous  cost  growth  between  their  inception  and  fielding. 
Attempts  to  understand  and  cope  with  cost  growth  date  back  at 
least  to  1962,  when  Peck  and  Scherer  published  one  of  the  first, 
important  studies  of  the  problem.*  Many  studies  have  followed. 
Congress  has  expressed  great  interest  in  the  subject  of  cost 
growth  and,  in  November  1981,  took  the  unprecedented  step  of 
requiring  that  Service  Secretaries  prepare  a  detailed  report  to 
Congress  if  cost  growth  on  a  major  program  exceeds  15%  or  face 
suspension  of  further  obligations.  If  cost  growth  exceeds  25%, 
the  Secretary  of  Defense  must  certify  to  Congress  that  the 
program  is  essential  and  that  the  new  cost  estimates  are 
reasonable.** 

Of  course,  cost  growth  can  occur  for  many  reasons  — some  of 
them  beyond  the  control  of  the  offerors  and  the  Government. 
These  reasons  include  the  many  risks  and  uncertainties 
associated  with  technically  advanced  and  complex  Defense 
systems.  Program  redirections,  scope  changes  and  quantity 
changes  also  cause  cost  growth  (or  may  be  caused  by  cost 
growth).  Some  causes,  such  as  inflationary  forces,  are  outside 
the  Defense  Department's  control. 

Cost  growth  may  also  be  due  to  an  award  to  an  offeror  with 
an  unrealistically  low  offer.  Such  unrealistic  offers,  especi¬ 
ally  for  RDT&E,  iray  be  due  to  overly  optimistic  projections  and 
competitive  pressures.  This  is  often  referred  to  as  the  'buy- 
in"  problem. 

When  an  offeror  who  is  unrealistically  low  wins  a  procure¬ 
ment,  the  likehood  of  a  cost  overrun  is  significantly  increased 


*  Merton  J.  Peck  and  Frederic  Scherer,  The  Weapons  Acqui¬ 
sition  Process:  An  Economic  Analysis.  Harvard  Univt-oity  iress, 
1962  . 

**  The  Nunn-McCurdy  Amendment  to  the  1983  DOD  Authorization 

Act. 
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and  probably  inevitable.  For  example,  if  the  Government  awards 
a  contract  to  an  offeror  whose  offer  is  20%  less  than  what 
appears  realistic,  there  is  a  good  likehood  that  there  will  be 
cost  growth  of  at  least  25  percent  (20/80)--so  due  to  this  lack 
of  cost  realism  alone  we  are  likely  to  reach  the  Nunn-McCurdy  25 
percent  Amendment  "threshold". 

Cost  growth  means  more  than  higher  costs;  it  may  lead  to 
schedule  slips,  to  program  cancellation,  and  even  to  the  cancel¬ 
lation  of  other  programs  in  order  to  provide  the  required 
additional  funds.  Cost  growth  may  also  require  drastic  changes 
in  system  design  and  program  scope.  Furthermore,  frequent  cost 
overruns  undermine  and  eventually  destroy  public  confidence  in 
the  Government's  capability  to  manage  its  programs. 

C.  COST  REALISM  ANALYSIS 

As  mentioned  in  the  Introduction,  data  have  been  collected 
on  proposed  costs  and  Government  estimates  for  eight  programs 
from  the  Naval  Electronic  Systems  Command  (NAVELEX).  This  data 
provides  some  insight  into  the  degree  of  realism  found  in 
contractor  cost  proposals.  In  developing  its  estimates,  NAVELEX 
evaluated  the  contractor  cost  proposals  in  detail,  required  the 
contractors  to  justify  their  data  and  estimating  rationale, 
developed  their  own  estimates  based  almost  entirely  on  the 
contractor's  data,  and  compared  these  estimates  with  the  cost 
proposals.  The  data  showed  the  following: 

Original  Offers  Best  &  Final  Offers 
(23  offers)  (19  offers) 

More  than  20%  low  30%  16% 

Within  3%  22%  47% 

One  out  of  three  original  offers  was  more  than  20  percent  low, 
and  one  out  of  six  best-and-f ina  1  offers  (BAFOs)  was  more  than 
20  percent  low.  Offers  more  than  20  percent  below  the  NAVELEX 
estimate  certainly  raise  a  question  as  to  cost  realism. 
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Interest : ng 1 y ,  less  than  one-quarter  of  the  original  offers  but 
nearly  half  of  the  BAFOs  were  within  3  percent  of  the  NAVELEX 
estimates,  apparently  as  a  result  of  the  detailed  Government 
cost  evaluation  and  the  fact  that  cost  realism  was  an  evaluation 
criterion . 

The  degree  of  realism  for  each  of  the  offerors  is  shown  in 
Figure  3.1.  Most  of  the  offers  are  lower  than  the  Government 
estimate.  The  BAFOs  are  noticeably  more  realistic.  Both  the 
offers  and  the  BAFOs  are  in  asending  order  of  realism.  The  BAFO 
corresponding  to  each  original  offer  is  shown  in  Figure  3.2. 
While  it  is  often  assumed  that  most  BAFOs  will  be  lower  than  the 
original  offers,  for  these  NAVELEX  programs  ,  11  of  the  18  BAFOs 
were  higher.  This  is  shown  in  Figure  3.3. 

D.  COST  REALISM  AS  REFLECTED  IN  GOVERNMENT  POLICY 

The  Federal  Acquistion  Regulations  (FAR),  formerly  the 
Defense  Acquisition  Regulations  (DAR),  is  the  primary  DOD 
document  establishing  procurement  policy.  While  the  FAR  does 
not  specifically  mention  "cost  realism",  it  does  refer  to  the 
reasonableness  of  costs  in  a  number  of  places.  It  states  that 
"fair  and  reasonable  prices"  are  to  be  negotiated,  that  "Govern¬ 
ment  procurement  is  co nc e r ne d  . . . w i t h  the  reasonableness  of  the 
price  which  the  Government  ultimately  pays"  and  that  "What  is 
reasonable  depends  on  a  variety  of  considerations  and  circum¬ 
stances."  The  FAR  also  states  that  "Proposals  for  cost  reim- 
burseable  type  or  fixed  price  contracts  may  be  penalized  during 
evaluation  to  the  degree  that  the  estimated  cost  is  unrealisti¬ 
cally  1  ow." 

DOD  issued  a  Directive  in  1976  concerning  source  selection 
(Directive  4105.62)  which  like  the  FAR  does  not  specifically 
mention  "cost  realism".*  The  Directive  does  refer  to  the 
reasonableness  and  credibility  of  costs.  It  states  that  pro¬ 
posals  should  "permit  verification  of  their  cost  credibility". 

*  Except  that  in  reference  to  discussions  with  the  contrac¬ 
tors  prior  to  selection,  the  Directive  states:  "The  discussion 
of  the  cost/price  proposals  may  i nc 1 u d e . . .  c o s t  realism  of  the 
offeror’s  proposal...." 
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It  also  states  that  "any  apparent  inconsistency  between  promised 
performance  and  cost  or  price  should  be  ex p 1  a i n ed . . .a n y  signi¬ 
ficant  inconsistency,  if  unexplained  raises  a  fundamental  issue 
of  the  competitor's  understanding  of  the  work  required. ..and  may 
be  grounds  for  rejection  of  the  proposal. ..burden  of  proof  of 
cost  credibility  rests  with  the  offeror".  Further,  the  Direc¬ 
tive  states  that  "The  primary  goal  of  evaluation  is  to  deter¬ 
mine,  for  each  competitor,  the  most  likely  outcome  in  terms  of 
system  characteristics,  costs  (development,  production  and 
support),  and  schedules". 

A  recent  draft  revision  of  DOD  Directive  4105.62  states 
that  "Independent  cost  estimates  are  necessary  as  a  bench-mark 
against  which  to  compare  proposed  cost  estimates...  The  realism 
of  the  offeror's  proposed  proposal  should  be  indicated  by  a 
ranking  relative  to  the  Government's  estimate." 

The  Government  Accounting  Office  issues  decisions  related 
to  source  selection  award  protests.  These  decisions  are  a  form 
of  Government  policy.  In  one  decision,  the  following  statement 
was  made  regarding  cost  realism:  "A  low  [offer]  should  not  be 
accepted  at  face  value...  An  agency  should  make  an  independent 
cost  project  ion. ..to  ensure  that  costs  are  examined  in  terms  of 
their  rea 1 ism. ..under  a  cost-reimbursement  type  contract...  [ GAO 
81-2  CPD  178]". 

None  of  the  Navy  acquisition  policy  documents  examined 
(NAVMAT  4200.49,  NAVELEX  4200. 12C,  and  NAVAIR  4200.24  and  .27) 
specifically  mention  "cost  realism".  However,  the  documents 
make  some  references  to  reasonableness  and  credibility  of  costs. 
For  example,  NAVMAT  Instruction  4200.49  requires  the  Contracting 
Officer  to  "assure  that  proposals  are  requested  on  a  basis  which 
permits  verification  of  their  cost  credibility." 

Both  Army  and  Air  Force  acquisition  policy  documents  refer 
to  "cost  realism"  in  some  detail.  The  Army  Materiel  Command 
(now  DARC0M)  issued  Regulation  715-1  in  1970  which  is  entitled 
"Cost  Realism  in  Proposal  Evaluations."  This  Regulation 
provides  the  following  definition:  "Cost  realism  is  defined  as 
the  employment  of  preplanned  methods  to  determine  the  probable 
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total  cost  for  a  procurement  at  completion;  cost  realism 
involves  a  comprehensive  analysis  to  develop  and  establish  the 
probable  overall  cost  of  performance  when  related  to  the 
required  technical  scope  of  the  work.”  The  purpose  of  the 
Regulation  is  "to  stress  cost  realism  procedures  as  a  method 
used  to  reduce  cost  growth  and  inaccurate  cost  estimating  on  the 
part  of  both  the  Government  and  the  potential  contractors." 
Cost  realism  is  to  be  used  "in  all  procurements  [involving] 
multiple  evaluation  factors...." 

The  Department  of  Air  Force  issued  Regulation  70-15  in  1976 
concerning  source  selection  which  states  that:  "Continued  effort 
must  be  exerted  to  achieve  greater  cost  realism  in  proposals  for 
major  development  programs."  It  also  states  that  "a  proposal 
may  be  penal ized... to  the  degree  that  the  proposed  cost/price  is 
unrealistically  low." 

A  number  of  excerpts  related  to  source  selection  cost 
evaluation  taken  from  the  above  policy  documents  are  provided  in 
Appendix  C. 

The  policy  documents  cited  above  provide  broad  guidance  in 
the  area  of  cost  evaluation  to  Government  personnel  conducting 
source  selections.  The  importance  of  assessing  the  reasonable¬ 
ness,  credibility,  validity,  and  realism  of  costs  is  stressed. 
While  the  term  cost  realism  does  not  appear  in  all  of  the  policy 
documents,  the  other  terms  used  have  approximately  the  same 
meaning.  The  policy  documents  are  not  very  specific  about  how 
to  assess  cost  realism  or  how  to  encourage  offerors  to  be  more 
realistic  in  their  cost  proposals. 
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SECTION  IV 
METHODOLOGY  SUMMARY 


The  source  selection  cost  evaluation  methodology  described 
in  this  Handbook  should  assure  a  sound  cost  evaluation  and 
should  encourage  contractors  to  make  more  realistic  offers. 
This  methodology  includes  four  interrelated  steps: 

(1)  Evaluation  Factors:  Determination  of  cost  eval¬ 
uation  factors; 

(2)  Solicitation:  Preparation  of  instructions  to  be 

included  in  the  solicitation  concerning  the  cost 
evaluation  factors; 

(3)  Government  Estimte:  Preparation  of  a  Government 
estimated  cost  for  each  contractor;  and 

(4)  Evaluation:  Rating  or  scoring  each  contractor  for 
Government  estimated  cost  and  cost  realism. 

The  methodology  involves  the  best  techniques  and  procedures 
currently  being  used  in  source  selection  cost  evaluation.  In 
addition,  this  Handbook  introduces  several  improvements  to  the 
current  techniques— most  importantly  techniques  for  numerical 
scoring  tradeoff  analyses. 


NOTE:  The  terms 

"Government 

estimated  cost" 

and 

"Government 

estimate”  are 

used  in  this  Handbook  for 

the 

Go  v  ernment 

estimate  of  the 

likely  cost 

for  each  contractor. 

The  methodology  associated  with  each  of  the  above  steps  is 
summarized  below  and  discussed  in  detail  in  the  next  four 
sections : 

Evaluation  Factors:  Make  cost  realism  as  well  as 
Government  estimated  cost  explicit  cost  evaluation 
sub-criteria.  Assess  cost  realism  based  on  the 
difference  between  the  contractor  offer  and  the 
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Government  estimate.  Assess  Government  estimated  cost 
•lased  on  the  likely  cost  to  the  Government  for  each 
icractor,  Make  Design-to-Unit-Production  Cost 
vli UPC)  and  Life  Cycle  Cost  (LCC)  sub-criteria  only  if 
■■edible  estimates  can  be  developed.  Even  then, 
generally  weight  DTUPC  and  LCC  low  relative  to  Govern- 
nt  estimated  cost  and  cost  realism. 

Solicitation:  Specify  cost  realism  in  addition 

o  Government  estimated  cost  as  cost  evaluation  sub- 
-iceria  in  the  solicitation  and  specify  the  Work 
eakdown  Struture  (WBS)  to  be  used  and  the  type  of 
backup  data  required.  The  WBS  should  be  tailored  to 
each  procurement  but  conform  as  much  as  practical  to 
•.,'BSs  for  similar  types  of  systems  and  programs. 

Government  Estimate:  Develop  a  Government 

•estimate  for  each  offeror  based  on  a  detailed  evalua- 
..‘.on  of  that  offeror's  proposal,  including  fact¬ 
finding  as  appropriate.  The  offerors  should  be 
-quired  to  justify  the  data  and  techniques  used  in 
developing  their  cost  proposals.  If  time  and 
;esources  permit,  estimates  based  on  parametric  and 
;;.;iiogy  techniques  should  also  be  made  as  a  cross- 
■  be k  . 

Evaluation:  Rate  or  score  Government  estimated 

cost  and  cost  realism  for  each  contractor.  If  quali¬ 
tative  rating  is  used,  apply  a  rating  or  grade  (e.g., 
an  adjective)  for  each  cost  evaluation  sub— criteria, 
if  numerical  scoring  is  used,  apply  a  triangle,  a 
trapezoid  or  normal  curve  scoring  rule.  The  Govern¬ 
ment  estimated  cost  score  is  a  function  of  the  differ¬ 
ence  between  the  Government  estimate  for  each  offeror 
and  the  lowest  Government  estimate  for  any  offeror, 
i’he  cost  realism  score  is  a  function  of  the  difference 
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between  the  offer  and  the  Government  estimate.  The 
differences  corresponding  to  a  minimum  score  (triangle 
or  trapezoid)  or  to  a  95  percent  likelihood  (normal 
curve)  should  be  determined  for  each  procurement  based 
on  sensitivity  and  tradeoff  analyses.  Also  prepare  a 
tabular,  and  possibly  graphical,  presentation  of  the 
a)  offers,  b)  Government  estimates,  c)  percent  differ¬ 
ences  between  the  Government  estimates  and  the  lowest 
Government  estimate,  and  d)  percent  differences 
between  the  offers  and  the  Government  estimates.  In 
addition,  prepare  a  narrative  cost  evaluation. 

It  is  believed  that  application  of  this  methodology  will 
encourage  contractors  to  make  more  realistic  offers  since  they 
will  be  on  notice,  though  the  solicitation,  that  cost  realism 
will  be  examined  and  could  be  a  deciding  factor  in  the  award. 
The  contractors  will  know  that  the  evaluation  procedures  will 
reduce  the  likelihood  of  awards  to  low  offers  on  cost  reimburs¬ 
able  contracts  unless  realistically  low  and  cost  is  the  deciding 
factor  . 

There  are  many  factors  involved  in  a  source  selection  and 
each  procurement  is  unique.  Therefore,  there  needs  to  be  flex¬ 
ibility  in  the  cost  evaluation  in  order  to  take  into  account 
such  factors  as  the  technical  and  program  risks,  the  stage  of 
de  v  e  1  opement ,  the  nature  of  the  hardware,  the  size  of  the 

procurement,  the  time  to  conduct  the  evaluation,  the  resources 
available  for  the  evaluation,  and  the  experience  of  the  source 
selection  team.  The  methodology  described  in  this  Handbook 
should  be  used  as  a  general  guide  and  applied  as  appropriate  to 
each  procurement. 

The  next  four  sections  discuss  each  of  the  above  steps  in 
detail.  An  illustrative  example  of  how  to  apply  the  methodology 
is  provided  in  Section  X. 
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SECTION  V 

EVALUATION  FACTORS 


Generally,  the  two  most  important  source  selection  factors 
or  criteria  are  technical  and  cost.  In  RDT&E  competitions, 
technical  is  usually  far  more  important  than  cost.  Additional 
evaluation  criteria  may  include  management,  supportability,  and 
schedule.  Each  evaluation  criterion  may  have  several  sub¬ 
criteria  . 

In  evaluating  cost,  the  most  important  factors  are  the 
likely  cost  to  the  Government  for  each  offeror  and  the  realism 
or  reasonableness  of  the  offers.  The  former  will  be  termed  the 
Government  estimated  cost  sub-criterion  and  the  latter  the  cost 
realism  sub-criterion.  Both  sub-criteria  are  important  because, 
from  the  cost  perspective,  the  award  of  a  contract  should  be 
made  not  just  on  the  basis  of  a  low  cost,  but  a  realistically 
low  cost  if  cost  is  the  deciding  factor.  However,  Government 
estimated  cost  will  generally  be  weighted  significantly  higher 
than  cost  realism,  in  order  to  avoid  trading  cost  realism  for  an 
unreasonably  high  price  (this  is  discussed  in  Section  IX, 
Sensitivity  and  Tradeoff  Analyses).  Other  cost  sub-criteria 
might  include  Design-to-Unit  Production  Cost  (DTUPC)  and/or  Life 
Cycle  Cost  (LCC). 

It  should  be  noted  that  the  Government  estimated  cost  for  a 
contractor  and  not  the  contractor's  offer  should  be  the  primary 
cost  evaluation  criterion.  This  should  help  to  assure  that 
offers  will  be  rea 1 istic--the  offeror  would  have  little  reason 
not  to  be  realistic  since  the  offer  is  used  only  in  figuring  the 
degree  of  cost  realism.  (Of  course,  the  offeror  might  decide 
not  to  be  realistic  in  certain  areas  which  he  figures  will  not 
be  discovered  during  the  Government's  cost  evaluation  and  fact¬ 
finding  .  ) 

In  a  sense,  all  cost  evaluations  will  address  the  realism 
or  reasonableness  of  contractors'  offers.  For  example,  cost 
evaluators  will  normally  check  for  completeness,  accuracy  and 
reasonableness  of  the  offers.  However,  if  the  contractors  are 
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to  be  encouraged  to  make  more  realistic  offers,  it  is  very 
important  to  make  cost  realism  an  explicit  evaluation  sub- 
criterion  even  if  a  small  weight  is  used.  In  this  way  the 
offerors  will  know  that  they  will  be  penalized  for  an  unrealis¬ 
tically  low  offer--a  "buy-in”  which  would  likely  lead  to  built 
in  cost  growth--and  penalized  for  an  unrealistically  high  offer 
which  would  likely  result  in  a  greater  cost  to  the  Government 
than  necessary.* 

Government  estimated  cost  can  be  assessed  on  the  basis  of 
the  likely  cost  to  the  Government  for  each  contractor.  The 
lower  the  Government  estimate,  the  better.  This  evaluation  sub¬ 
criterion  also  requires  that  a  realistic  Government  estimate  be 
developed,  as  described  in  Section  VII. 

Cost  realism  can  be  assessed  on  the  basis  of  the 
difference  between  the  proposed  cost  (the  offer)  and  a  Govern¬ 
ment  estimate  of  the  realistic  cost  for  that  contractor.  The 
smaller  this  difference,  the  more  realistic  the  offer.  There¬ 
fore,  a  realistic  Government  estimate  for  each  contractor  is  of 
critical  importance. 

DTUPC  and  LCC  may  be  important  in  some  RDT&E  competi tions. 
For  example,  because  of  different  designs  and  approaches  by 
different  contractors,  there  may  be  significant  differences  in 
DTUPC  and  LCC.  And,  procurement  and  operations  savings  may  more 
than  offset  higher  development  costs.  Nevertheless,  DTUPC  and 
LCC  should  generally  be  given  relatively  little  weight  compared 
to  Government  estimated  cost  and  cost  realism.  This  will  assure 
that  Government  estimated  cost  and  cost  realism  have  sufficient 
weight  in  the  overall  source  selection  evaluation.  Too  many 
sub-criteria  should  be  avoided  because  the  effect  of  each  gets 
"watered  down"  in  terms  of  overall  score.  Furthermore,  DTUPC 
and  LCC  may  be  difficult  to  estimate  early  in  an  RDT&E  program. 


*  Under  a  cost  reimbursable  contract,  the  contractor  is 
paid  only  for  costs  incurred  plus  a  fee.  But,  if  he  is  given  an 
award  for  an  amount  substantially  higher  than  the  Government 
estimated  cost  (i.e.,  more  than  is  realistic),  then  he  will 
generally  find  ways  to  spend  the  extra  money--if  not  conscious¬ 
ly,  then  through  less  incentive  to  be  efficient. 
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and  there  may  be  a  considerable  expense  to  contractors  and  the 
Government  associated  with  evaluating  DTUPC  and  LCC.  Also, 
contractors  are  generally  not  held  accountable  for  their  DTUPC 
and  LCC  estimates. 

Prior  to  issuing  the  solicitation,  weights  for  each  of  the 
criteria  and  sub-criteria  should  be  determined.  For  RDT&E 
programs,  cost  will  usually  be  weighted  much  lower  than 
technical.  For  example,  cost  might  be  20  to  30  percent  and 
technical  50  to  70  percent  of  the  total  score,  with  other 
criteria  making  up  the  difference.  Deciding  on  the  weighting  of 
cost  and  the  breakout  between  Government  estimated  cost  and  cost 
realism  should  be  based  on  experience.  If  numerical  scoring  is 
to  be  used,  an  analysis  can  be  made  of  the  effects  of  different 
weights  and  scoring  rules  on  the  overall  scores  for  hypothetical 
proposals.  Such  an  analysis  should  consider  the  tradeoff 
between  cost  and  technical  scores  (e.g.,  the  degree  of  cost 
realism  that  would  be  traded  for  a  certain  degree  of  technical 
superiority).  This  is  discussed  further  in  Section  IX,  Sensi¬ 
tivity  and  Tradeoff  Analyses. 


Methodology:  Make  cost  realism  as  well  as  Government  esti¬ 
mated  cost  explicit  cost  evaluation  sub-criteria.  Assess 
cost  realism  based  on  the  difference  between  the  contractor 
offer  and  the  Government  estimate.  Assess  Government 
estimated  cost  based  on  the  likely  cost  to  the  Government 
for  each  contractor.  Make  DTUPC  and  LCC  sub-criteria  only 
if  credible  estimates  can  be  developed.  Even  then,  general¬ 
ly  weight  DTUPC  and  LCC  low  relative  to  Government  estimated 
cost  and  cost  realism. 


Making  cost  realism  an  explicit  evaluation  criterion  has 
great  merit.  It  places  contractors  on  notice  that  cost  realism 
will  be  examined  and  could  be  a  deciding  factor  in  the  award. 
The  Government  source  selection  team  is  also  on  notice  to 
address  cost  realism  specifically.  The  actual  evaluation  of 
cost  realism  can  be  done  at  many  levels  of  detail  depending  on 
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the  procurement  and  the  personnel  available.  Regardless  of  the 
level  of  detail,  the  key  point  is  to  make  cost  realism  an 
evaluation  factor  which  the  contractors  know  can  influence  the 
award.  Furthermore,  a  detailed  cost  evaluation,  such  as  that 
required  for  assessing  cost  realism  can  help  in  negotiation  and 
is  likely  to  result  in  best-and-f ina 1  offers  which  are  more 
realistic  than  the  original  offer. 


SECTION  VI 
SOLICITATION 


A.  GENERAL 

The  solicitation,  in  the  form  of  an  RFP  or  R  F  Q ,  is  the 
formal  document  specifying  the  Government's  requirements 
associated  with  an  impending  procurement,  as  well  as  such 
matters  as  the  terms  anil  conditions  and  evaluation  criteria.  It 
is  important  to  inform  the  prospective  offerors  in  the  solici¬ 
tation  that,  among  other  factors,  they  will  be  evaluated  on  the 
basis  of  cost  realism  and  that,  an  unrealistic  offer  may  be 
grounds  for  rejecting  the  proposal.  Knowing  that  they  may  be 
penalized  for  unrealistic  offers  will  certainly  encourage  the 
offerors  to  be  more  realistic.  Whether  they  will  in  fact  be 
more  realistic  depends  on  how  the  offerors  weigh  the  penalty  foi 
being  unrealistic  against  other  factors,  such  as  a  belief  that 
the  award  will  be  biased  toward  the  low  offer  despite  any 
unrealism . 

Since  the  cost  proposals  must  be  evaluated  by  the  Govern¬ 
ment  in  some  detail  in  order  to  determine  whether  they  are 
realistic  and  complete,  it  is  important  that  the  solicitation 
specify  a  work  breakdown  structure  (WBS)  tailored  to  the 
procurement  and  specify  the  type  of  data  required  to  backup  the 
cost  proposal.  By  specifying  a  WBS,  consistency  among  offerors 
is  increased  and  the  WBS  can  be  tailored  to  break  out  those 
areas  in  which  cost  visibility  is  desired,  such  as  areas  where 
risk  is  considered  greatest.  Thus,  a  well  structured  WBS  can 
simplify  the  job  of  cost  evaluation. 

It  is  desirable  that  WBSs  become  more  standardized  and 
consistent  for  all  programs  so  that  improved  cost  data  bases  can 
be  developed  which  will  facilitate  future  cost  estimating  and 
cost  evaluation.  Achieving  more  standardized  and  consistent 
WBSs  requires  more  commonality  of  WBS  items  at  the  higher  levels 
of  the  WBS  structure  for  all  programs  than  is  now  the  case.  It 
also  requires  nearly  identical  WBSs  except  at  the  lowest  levels 
of  the  WBS  structure  for  similar  types  of  systems  and  programs. 


While  Military  Standard  881A  has  helped  to  standardize  WBSs,  it 
is  somewhat  general  and  allows  a  lot  of  latitude  in  structuring 
a  WBS.  The  Office  of  the  Secretary  of  Defense  and  the  higher 
levels  of  the  Services  have  the  primary  responsibility  for 
achieving  greater  standardization  and  consistency  of  WBSs,  but 
each  source  selection  organization  can  help  by  conforming  as 
much  as  practical  to  WBSs  for  similar  types  of  systems  and 
programs.  Improved  WBS  formats  would  be  particularly  helpful  in 
developing  far  more  consistent  and  useful  cost  data  bases  from 
actual  contract  costs.  Such  improved  data  bases  would  facili¬ 
tate  not  only  future  cost  estimating  but  also  contract  monitor¬ 
ing  . 

By  carefully  specifying  the  type  of  cost  proposal  backup 
data,  the  job  of  cost  evaluation  can  also  be  greatly  simplified. 
Examples  of  some  of  the  backup  data  that  might  be  requested  are: 
a  list  of  material  high  dollar  items;  a  list  of  all  tools,  test 
equipment  and  facilities;  and  monthly  manloading  charts  by 
functional  labor  categories. 

It  should  be  noted  that  under  Public  Law  98-369,  known  as 
the  Truth  in  Negotiations  Act,  contracting  officers  are  required 
to  obtain  cost  or  pricing  data  from  contractors  for  negotiated 
procurements  exceeding  $100,000  (recently  revised  from  the 
$500,000  threshold  in  Public  Law  87-653).  In  submitting  such 
data,  each  contractor  must  certify  that,  to  the  best  of  his 
knowledge  and  belief,  the  data  are  accurate,  complete,  and 
current.  The  certification  applies  to  the  validity  of  factual 
data  in  the  contractor’s  possession,  and  not  to  the  efficacy  of 
estimates  or  judgment  factors.  The  contractor  is  only  required 
to  submit  data  that  are  "reasonably  available"  to  him. 

Therefore,  the  Government  has  a  legal  right  to  detailed 
cost  and  pricing  data.  Furthermore,  D0D  Directive  4105.62 
states  that  the  "burden  of  proof  of  cost  credibility  rests  with 
the  offeror." 
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Methodology:  Specify  cost  realism  in  addition  to  Government 
estimated  cost  as  cost  evaluation  sub-criteria  in  the 
solicitation  and  specify  the  WBS  to  be  used  and  the  type  of 
backup  data  required.  The  WBS  should  be  tailored  to  each 
procurement  but  conform  as  much  as  practical  to  WBSs  for 
similar  types  of  systems  and  programs. 


Specific  clauses  that  can  be  used  in  the  solicitation  are 
suggested  below. 

B.  SPECIFIC  CLAUSES 

Various  clauses  should  be  included  in  the  solicitation 
which  clearly  state  the  Government  intentions  regarding  the 
evaluation  of  costs.  Suggested  clauses  are  provided  below 
related  to  the  cost  evaluation  in  general,  the  cost  evaluation 
criteria,  and  the  cost  proposal  data  required  (including  the  WBS 
and  back-up  data).  These  clauses  are  based  largely  on  clauses 
developed  by  NAVELEX.  The  cost  evaluation  team  should  tailor 
the  exact  wording  of  these  clauses  for  each  program.  Placement 
of  these  clauses  within  the  solicitation  will  depend  upon  the 
organization  of  the  RFP.  A  key  point  i3  to  explicitly  inform 
the  offerors  that  cost  realism  will  be  examined  and  could  be  a 
deciding  factor  in  the  award.  The  words  or  phrases  in  brackets 
are  optional  or  may  be  modified  to  reflect  the  specific  program. 


General  Instructions 

The  solicitaion  must  inform  the  offerors  of  the  basis  for 
the  award--the  evaluation  criteria  and  their  importance.  The 
following  clauses  may  be  used: 

Proposals  received  in  response  to  this  solici- 
tion  will  be  evaluated  by  the  Government  [NAVAIR, 
NAVELEX,  etc.]  pursuant  to  Federal  Acquisition 
Regulations.  A  contractor  will  be  selected  on  the 
basis  of  his  proposal  being  most  advantageous  to  the 
Government,  technical,  cost  and  management  factors 
considered  . 

Primary  emphasis  will  be  placed  on  the  technical 
and  cost  factors;  however,  the  technical  factor  will 
receive  [more]  emphasis  than  the  cost  factor.  Within 
the  technical  area....  Within  the  cost  area,  Govern- 
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ment  estimated  cost  will  receive  [more]  emphasis  than 
cost  realism. 

Offerors  should  also  be  informed  that  the  source  selection 
evaluation  will  address  the  realism  or  reasonableness  of  the 
their  offers.  The  following  clauses  may  be  used: 

An  offeror's  proposal  is  presumed  to  represent 
the  best  efforts  to  respond  to  the  solicitation;  Any 
inconsistency,  whether  real  or  apparent,  between 
promised  performance  and  cost  should  be  explained  in 
the  proposal.  For  example,  if  the  intended  use  of  new 
and  innovative  production  techniques  is  the  basis  for 
an  abnormally  low  estimate,  the  nature  of  these  tech¬ 
niques  and  their  impact  on  cost  should  be  explained; 
or,  if  a  corporate  policy  decision  has  been  made  to 
absorb  a  portion  of  the  estimated  cost,  that  should  be 
stated  in  the  proposal.  Any  significant  inconsistency, 
if  unexplained,  raises  a  fundamental  issue  of  the 
offeror's  understanding  of  the  nature  and  scope  of  the 
work  required  and  of  the  ability  to  perform  the 
contract,  and  may  be  grounds  for  rejection  of  the 
proposal.  The  burden  of  proof  as  to  cost  credibility 
rests  with  the  offeror. 

Cost  Evaluation  Criteria 

The  solicitation  should  specify  that  both  Government  esti¬ 
mated  cost  and  cost  realism  will  be  cost  evaluation  criteria. 
The  following  clauses  may  be  used: 

The  cost  evaluation  shall  consider  and  assess  the 
most  likely  cost  to  the  Government  (including  fee)  and 
the  realism  or  reasonableness  of  the  offers.  [  The 
cost  evaluation  shall  also  consider  and  assess  DTUPC 
and  LCC.]  The  former  is  termed  the  Government  esti¬ 
mated  cost  sub-cr itrion ,  and  the  latter  is  termed  the 
cost  realism  s u b -c r i t e r i o n .  The  award  of  a  contract 
will  be  made  not  just  on  the  basis  of  a  low  cost,  but 
a  realistically  low  cost.  Both  the  Government  esti¬ 
mated  cost  and  the  cost  realism  criteria  involve  the 
development  by  the  Government  of  a  realistic  estimate 
for  each  offeror.  [This  estimate  will  be  based  on  the 
offeror's  design,  a  review  of  the  tasks  required  to 
accomplish  this  effort,  a  determination  of  the 
required  manpower  and  materials  for  each  WBS  element, 
and  offeror's  historical  cost  as  demonstrated  by  the 
offeror.  To  assist  with  the  Government  evaluation, 
offerors  are  required  to  furnish  the  procedures  and 
rationale  used  in  compiling  their  proposed  costs.  All 
information  such  as  IR&D  efforts,  etc.,  which  an 
offeror  wants  the  Government  to  consider  in  the  cost 
evaluation  must  be  disclosed.] 
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The  cost  evaluation  criteria  are  defined  as 
follows : 

Government  Estimated  Cost.  Government  estimated 
cost  will  be  assessed  on  the  basis  of  the  Government 
estimate  of  the  realistic  cost  for  each  offeror.  This 
estimate  for  each  offeror  will  be  evaluated  [and 
scored  numerically]  to  determine  its  comparative 
advantage  to  the  Government.  The  lower  the  Government 
estimate,  the  better  [or,  the  higher  the  score]. 

Cost  Realism.  The  degree  of  cost  realism  in  the 
offeror's  cost  proposal  will  be  assessed  in  general  by 
checking  the  completeness,  accuracy,  d e f en s i b i 1 i t y , 
and  reasonableness  of  the  proposal.  Cost  realism  will 
also  be  assessed  [and  scored  numerically]  on  the  basis 
of  the  difference  between  the  proposal  (the  bid  or 
offer)  and  the  Government  estimate  of  the  realistic 
cost  for  that  offeror.  The  smaller  this  difference, 
the  more  realistic  the  offeror's  proposed  cost  [and 
the  higher  the  score].  An  excessively  high  or  low  cost 
will  indicate  the  offeror's  lack  of  understanding  of 
the  requirments.  Proposals  may  be  penalized  to  the 
degree  that  the  proposed  costs  are  unrealistic. 

Cost  Proposal  Data  Requirements 

In  addition  to  mentioning  the  cost  evaluation  criteria,  the 
solicitation  should  specify  the  WBS  to  be  used  and  the  type  of 
back-up  data  required.  By  carefuly  developing  the  WBS  and 
specifying  the  type  of  back-up  data  needed,  the  cost  evaluation 
task,  including  development  of  a  Government  estimate,  is  made 
easier.  The  following  clauses  may  be  used: 

The  cost  proposal  shall  be  prepared  using  the 
work  breakdown  structure  specified  in  Table  X. 
Explain,  in  whatever  detail  is  required,  the  method¬ 
ology  used  to  estimate  each  element  of  cost  (e.g., 
labor,  material,  etc.)  for  each  WBS  element. 

In  all  cases  where  cost  estimates  are  based  upon 
past  experience,  the  offeror  shall  identify  the  past 
experience,  explain  how  the  past  experience  relates  to 
the  current  effort,  and  explain  how  cost  data  avail¬ 
able  from  the  past  experience  were  adapted  to  the 
current  effort.  If  the  past  experience  concerns  a 
specific  hardware  item  built  or  acquired  in  the  past, 
the  offeror  shall  identify  the  item,  applicable  dates 
and  item  cost. 

In  all  cases  where  cost  estimates  are  based  upon 
learning/improvement  curve  applications,  the  offeror 
should  identify  the  specific  area  subject  to  learning, 
the  curve  hypothesis  (unit  or  cumulative)  and  the 
slope  of  the  curve  as  a  percent.  Also,  the  offeror 
shall  explain  what  data  were  used  to  develop  the 
slope,  how  these  data  relate  to  the  current  effort  and 
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how  entry  into  the  learning  curve  was  attained,  (e.g. 
how  the  first  unit  was  derived). 

Monthly  manloading  charts  shall  be  provided  for 
each  WBS  element.  Also,  monthly  manloading  charts 
shall  be  provided  by  functional  labor  category  (i.e., 
engineering,  manufacturing,  management,  etc.)  for  the 
total  effort  [or  for  level  X  WBS  elements], 

A  chart  shall  be  submitted  which  summarizes 
engineering,  manufacturing  and  other  direct  labor 
category  hourly  rates.  In  addition,  a  chart  shall  be 
submitted  which  summarizes  engineering,  manufacturing, 
material  handling,  G&A,  and  other  burden  rates  by  year 
for  the  applicable  years  of  the  contract. 

The  offeror  shall  indicate  whether  labor  and 
burden  rates  are  Government  approved  bidding  rates.  If 
not,  indicate  the  basis  on  which  rates  were 
established . 

For  all  material  items  except  major  subcontracts 
and  in  ter d i v isiona 1  transfers,  provide  a  list  of  the 
high  dollar  items  to  cover  approximately  80  percent  of 
the  total  material  cost.  The  list  shall  include  the 
following:  identification  of  the  part(s);  WBS  element; 
word  description  [e.g.,  power  supply,  IC,  capaci¬ 
tor];  quantity  required  by  WBS  element;  total  quantity 
for  the  contract;  unit  cost;  total  or  extended  cost; 
and  the  basis  of  the  estimate.  [A  trace  between  the 
list  and  the  price  on  the  DD-633  shall  be  provided. 
This  trace  shall  identify  all  factors,  cost  of  each 
factor,  and  the  method  for  estimating  each,  such  as 
attrition,  freight,  variance,  etc.  The  rationale  for 
estimating  material  costs  and  choosing  material 
sources  shall  be  explained  including  a  percentage 
breakout  (by  dollar  value)  of  those  based  on  quotes, 
engineering  estimates,  historical  costs  from  previous 
procurements,  etc.] 

The  offerors  shall  provide  similar  information  as 
discussed  above  (i.e.,  costs,  manloading  charts, 
burden  rates,  material  costs)  for  each  of  the  major 
subcontractors. 

The  offerors  shall  provide  a  list  of  all  tools, 
test  equipment  and  facilities  to  perform  the  effort 
[except  for  those  which  will  be  provided  as  capital]. 
The  estimated  dollar  value  of  each  item  shall  be 
provided  . 

Computer  programming  (software)  cost  estimates 
shall  include  the  method  used  to  derive  the  cost 
estimate  and  the  source  of  the  information.  A  table 
shall  be  provided  which  itemizes  the  number  of 
instructions  per  module  and  the  estimated  number  of 
man-months  per  module.  The  estimate  shall  include  all 
efforts  from  design  through  coding,  debugging,  and 
system  test . 
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Best  and  Final  Offer  (BAFO)  Instructions 

After  the  proposals  have  been  received  and  reviewed  by  the 
Government,  and  discussions/negotiations  completed  with  offerors 
within  the  competitive  range,  the  offerors  are  often  provided  an 
opportunity  to  submit  proposal  revisions  (i.e.,  a  BAFO). 
Instructions  to  the  offerors  submitting  BAFOs  should  reiterate 
the  importance  of  cost  realism  by  including  a  clause  such  as  the 
following : 

The  offeror  must  fully  substantiate  in  the  best- 
and-final  offer  any  cost/price  changes  made  to  the 
original  proposal.  Lump  sum  reductions  in  cost/price 
will  not  be  accepted  without  supporting  rationale. 
Supporting  data  must  provide  traceability  to  the 
technical,  business,  or  financial  conditions  that 
caused  the  change. 
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SECTION  VII 
GOVERNMENT  ESTIMATE 


The  key  to  assessing  the  realism  and  reasonableness  of  cost 
proposals  is  the  development  of  a  realistic  Government  estimate 
for  each  offeror,  which  can  be  compared  against  that  offeror’s 
proposed  cost.  While  a  cost  proposal  can  be  analyzed  and  its 
realism  judged  on  a  piecemeal  basis  (e.g.,  examining  a  few  major 
cost  elements  and  commenting  on  the  degree  of  realism),  such  an 
approach  is  generally  not  sufficient.  The  Government  should 
make  an  estimate  for  each  offer  of  the  most  likely  cost  that 
will  be  incurred  as  a  result  of  awarding  a  contract  to  that 
offeror.  An  estimate  of  the  most  likely  cost  for  each  offeror 
is  termed  the  "Government  estimated  cost"  or  "Government  esti¬ 
mate"  in  this  Handbook.  The  Government  estimate  for  each  offer¬ 
or  and  the  difference  between  the  offer  and  the  Government 
estimate  for  each  offeror  (i.e.,  the  degree  of  realism)  are  the 
principal  cost  evaluation  sub-criteria  as  discussed  in  Section 
V.  Therefore,  it  is  important  that  the  Government  estimates  be 
realistic  and  reasonable. 

The  Government  estimates  should  be  based  on  a  detailed 
evaluation  of  each  offeror’s  proposal  including  the  offeror's 
data,  history  and  estimating  techniques.  The  offerors  should  be 
required  to  justify  the  data,  history  and  estimating  techniques 
used.  Both  direct  and  indirect  costs  should  be  evaluated. 
Direct  costs  are  direct  labor,  direct  material,  and  other  direct 
costs  which  are  specifically  and  uniquely  attributable  to  a 
particular  product  or  service.  Indirect  costs  are  overhead,  G&A 
and  fee.  Direct  costs  are  the  "nucleus"  around  which  total 
contract  price  is  built  since  the  indirect  costs  are  generally  a 
function  of  direct  costs. 

The  Federal  Acquisition  Regulations  (FAR)  define  the  type 

of  cost  analysis  and  evaluation  that  is  required  for  cost- 

reimbursable  contracts  as  follows: 

The  review  and  evaluation  of  a  contractor’s  cost  or 
pricing  data  and  of  the  judgmental  factors  applied  in 
projecting  from  the  data  to  the  estimated  costs  in 
order  to  form  an  opinion  leading  to  a  position  on  the 
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degree  to  which  the  contractor’s  proposed  costs  repre¬ 
sent  what  contract  performance  should  cost,  assuming 
reasonable  economy  and  efficiency...  It  includes 
appropriate  verification  of  cost  data,  evaluation  of 
specific  elements  of  costs  and  projection  of  these 
data  to  determine  the  effect  on  price  factors  like 
cost  necessity,  allowance  for  contingencies,  and  the 
basis  used  for  allocation  of  overhead  costs. 

The  Government  estimates  need  to  be  specific  to  each 
offeror  as  stated  above.  This  is  important  because  the  costs 
likely  to  be  incurred  by  each  contractor  may  be  different, 
sometimes  significantly  so,  even  though  each  contractor  is 
expected  to  perform  to  the  same  set  of  requirements  and  state¬ 
ment  of  work.  Some  of  the  reasons  that  the  costs  differ  among 
contractors  are  as  follows.  Labor  rates  differ  due  to  geo¬ 
graphical  differences  and  labor  mix  differences.  Overhead  rates 
vary  for  many  reasons  including  the  projected  business  base  of 
each  contractor.  Accounting  systems  differ  including  the 
classification  of  direct  and  indirect  costs  and  the  manner  of 
distributing  indirect  costs.  Facilities  and  capital  equipment 
differ  including  the  degree  of  automation  (e.g.,  computer-aided- 
design  (CAD)  and  computed-aided-manuf actur ing  (CAM)).  Manufac¬ 
turing  processes  differ.  Previous  experience  differs  including 
related,  relevant  engineering  drawings,  "breadboards",  and  test 
equipment.  Some  contractors  may  have  already  performed  relevant 
IR&D  work. 

In  some  cases  a  contractor’s  data  and  history  may  not  apply 
or  may  not  be  appropriate  in  evaluating  certain  portions  of  a 
cost  proposal.  In  these  cases,  parametric,  analogy  and  other 
techniques  of  cost  analysis  which  utilize  cost  data  from  a 
number  of  sources  may  be  used.  Estimates  developed  using  there 
techniques  may  sometimes  be  adjusted  to  reflect  differences 
between  contractors.  These  techniques  may  also  be  used  to 
cross-check  estimates  which  are  based  on  specific  contractor 
cost  data. 

Parametric  techniques  involve  the  development  and  appli¬ 
cation  of  cost  estimating  relationships  (CERs)  based  on  the 
relationship  between  historical  costs  for  systems  or  components 


7-2 


of  systems  and  various  technical,  performance  and  other  charac¬ 
teristics.  Analogy  techniques  involve  extrapolations  from 
actual  costs  for  similar  systems.  Other  techniques  may  include 
the  use  of  industry  wide  factors.  Parametric  and  analogy 
techniques  are  the  methods  used  for  developing  generic  estimates 
of  the  type  often  used  for  budgetary  estimates  and  are  generally 
applied  at  a  relatively  aggregated  level  of  the  WBS. 

A  recommended  methodology  for  developing  a  Government 
estimated  cost  for  each  offeror  is  as  follows.  The  first  step 
is  an  examination  of  the  cost  proposals  and  any  backup  data  for 
completeness  and  consistency.  Then,  extensive  "fact-finding"  is 
conducted  at  the  offerors’  plants,  where  clarifications  can  be 
obtained,  additional  backup  data  examined,  and  the  offerors 
asked  to  justify  their  cost  estimates  based  on  their  actual 
experience  on  previous  programs.  Through  such  fact-finding,  the 
validity  and  applicability  of  the  offerors’  data  and  estimating 
techniques  can  be  determined.  Alternatively,  the  offerors  may 
be  invited  to  Government  offices  to  answer  questions,  but 
without  access  to  their  complete  data  bases  the  offerors  are 
limited  in  the  answers  they  can  provide.  For  direct  labor,  some 
of  the  things  examined  are  labor  standards  for  repetitive  type 
operations  and  manloading  charts  as  a  function  of  time  for 
engineering  and  support  activities.  For  direct  materials,  some 
of  the  things  examined  are  the  bill  of  materials  and  shrinkage 
factors  (material  lost  during  fabrication  due  to  scrap,  spoilage 
and  obsolescence).  In  addition,  labor  rates,  overhead  rates, 
G&A  rates,  and  vendor  quotes  are  examined,  generally  by  the 
Defense  Contracts  Audit  Agency  (DCAA).  Based  on  this  detailed 
analysis  of  direct  and  indirect  costs,  a  realistic  Government 
estimate  is  then  prepared  for  each  line  of  the  WBS — accepting 
the  contractor's  estimate  as  is  for  that  line  or  adjusting  the 
contractor’s  estimate  based  on  the  analysis  of  the  contractor's 
history,  on  the  correction  of  any  errors  found,  and  on  the  use 
of  the  Government's  data  base  for  other  programs  when  the  con¬ 
tractor’s  history  is  lacking  or  not  relevant. 
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The  process  of  developing  a  Government  estimated  cost  for 
each  offeror  is  further  described  in  the  "Armed  Services 
Procurement  Regulation  Manual  for  Contract  Pricing"*  under  the 
subject  "contract  cost  analysis".  The  process  is  also  described 
in  considerable  detail  in  a  Navy  guide,  "An  Introduction  to 
Direct  Cost  Analysis,"**  which  is  the  basis  for  a  one-week 
course  taught  regularly  at  the  Navy  Acquistion/Logistics  Manage¬ 
ment  Training  Center  in  Washington,  D.C.,  and  at  other  locations 
from  time  to  time.  The  guide  and  the  course  were  largely 
developed  by  Irwin  Neveleff  of  NAVELEX. 

In  developing  a  Government  estimated  cost,  a  well 
structured  WBS  and  a  careful,  thoughtful  specification  for  cost 
backup  data  in  the  solicitation  can  make  the  task  easier  for  the 
cost  evaluation  team,  as  discussed  in  the  previous  section. 

The  parametric  and  analogy  techniques  discussed  above  may 
be  used  to  develop  estimates  of  a  program  prior  to  receipt  of 
the  contractors*  proposals.  These  estimates  can  then  be 
compared  with  the  contractors*  proposals  to  indicate  areas  where 
the  detailed  fact-finding  should  be  directed  (i.e.,  areas  that 
have  the  greatest  potential  payoff).  This  could  reduce  the 
amount  of  analysis  and  time  required  (and  presumably  personnel 
resources  required)  during  the  detailed  fact-finding.  Some¬ 
times,  cost  estimates  will  already  have  been  prepared  for  budget 
purposes  or  program  review  purposes.  These  estimates  may  only 
require  updating  and  tailoring  to  the  specific  requirements  of 
the  solicitation. 

Technical  risks  must  be  considered  in  preparing  the  Govern¬ 
ment  estimates.  Engineers  from  the  program  office,  source 
selection  organization  and  the  Defense  Contract  Administration 
Service  (DCAS)  may  assist  in  the  risk  assessment  and  cost  eval¬ 
uation.  It  also  helps  if  some  of  the  cost  analysts  preparing 
the  Government  estimates  have  an  engineering  background. 


*  Armed  Services  Procurement  Regulation  Manual  for 
Contract  Pricing,  ASPM  No.  1,  edition  of  15  September  1975. 

**  An  Introduction  to  Direct  Cost  Analysis:  A  Basic  "How¬ 
to"  Guide  for  NAVSEC  F-COG  and  S-COG  Negotiatied  Production 
Procurements .  Revision  1,  2  July  1973. 


Since  there  are  technical  and  program  (e.g.,  schedule) 
risks  and  uncertainties  associated  with  any  program,  as  well  as 
uncertainites  in  cost  estimating  data  and  techniques,  there  is 
often  considerable  uncertainty  in  the  Government  estimates. 
Therefore,  it  is  important  to  take  into  account  the  uncertainty 
associated  with  the  Government  estimates  when  judging  the 
realism  of  the  offerors'  proposed  costs.  The  greater  the  degree 
of  uncertainty  in  a  Government  estimate,  the  more  admissible  are 
greater  differences  between  the  proposed  cost  and  the  Government 
estimate.  To  provide  an  indication  of  uncertainty,  it  may  be 
desirable  to  estimate  a  range  of  costs  for  each  WBS  item,  rather 
than  just  a  point  (most  likely)  estimate.  These  ranges  and  most 
likely  estimates  can  then  be  used  to  characterize  the  cost 
uncertainty.  However,  because  there  is  a  great  deal  of  sub¬ 
jectivity  in  establishing  such  ranges  and  because  certain  risks 
and  uncertainties  may  not  even  be  considered,  it  is  difficult  to 
obtain  a  clear,  consistent  characterization  of  the  cost  uncer¬ 
tainty.  This  is  further  complicated  in  that  some  risks  and 
uncertainties  (such  as  scope  changes)  should  not  be  taken  into 
account  in  evaluating  cost  proposals.  Cost  uncertainty  is 
discussed  further  in  the  next  section. 

It  is  also  important  to  achieve  consistency  in  the  Govern¬ 
ment  estimates  for  different  offerors.  All  offerors  are  propos¬ 
ing  to  the  same  requirements,  but  their  approaches  will  differ 
and  the  requirements  will  not  be  specific  in  all  areas.  There¬ 
fore,  different  conclusions  will  be  reached  regarding  such 
things  as  the  number  of  tests  required  of  each  type.  In  some 
cases,  these  differences  between  offerors  may  be  reasonable 
because,  for  example,  offerors  may  use  different  design 
approachs  and  will  have  different  prior  design  and  testing 
experience.  In  other  cases,  the  differences  may  not  be  reason¬ 
able  because,  for  example,  the  number  of  flight  tests  finally 
negotiated  might  be  the  same  regardless  of  offeror.  Therefore, 
in  some  cases,  the  cost  proposals  should  be  put  on  a  common 
basis.  This  is  more  important  for  evaluating  Government  esti¬ 
mated  cost  than  cost  realism  since  cost  realism  is  evaluated 


7-5 


relative  to  the  Government  estimated  cost  which,  for  any 
offeror,  will  be  on  a  common  basis  with  the  cost  proposal  for 
that  offeror. 


Methodology:  Develop  a  Government  estimate  for  each  offeror 
based  on  a  detailed  evaluation  of  that  offeror's  proposal, 
including  fact-finding  as  appropriate.  The  offerors  should 
be  required  to  justify  the  data  and  techniques  used  in 
developing  their  cost  proposals.  If  time  and  resources 
permit,  estimates  based  on  parametric  and  analogy  techniques 
should  also  be  made  as  a  cross-check. 
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SECTION  VIII 
EVALUATION 


A.  EVALUATION  ALTERNATIVES 

Considerable  information  is  developed  on  the  strengths  and 
weaknesses  of  each  of  the  offerors'  propolis  during  the  source 
selection  evaluation  of  technical,  cost  and  other  factors.  This 
information  needs  to  be  summarized  and  structured  in  such  a  way 
as  to  provide  the  Source  Selection  Authority  (SSA)  with  a  sound 
basis  for  making  an  award.  Generally,  the  offerors  are  rated  or 
scored  for  each  of  the  evaluation  sub-criteria,  which  are  then 
weighted  to  produce  an  overall  rating  or  score.  The  evaluators 
may  assign  numerical  scores  (e.g.,  zero  to  10)  or  adjectival 
ratings  (e.g.,  excellent,  unsatisfactory,  etc).  Sometimes,  the 
adjectival  ratings  are  converted  to  numerical  scores  so  that  an 
overall  score  can  be  developed.  A  variation  of  adjectival 
rating  is  color  rating,  in  which  colors  represent  various 
degrees  of  acceptability  (e.g.,  blue  for  exceptional).  In  some 
cases,  the  evaluation  sub-criteria — particularly  those  related 
to  cost  — may  not  be  rated  or  scored  at  all.  However,  in  such 
cases  a  narrative  evaluation  would  generally  be  provided.  Also, 
tablular  and  graphical  displays  may  be  useful. 

These  various  evaluation  alternatives  are  summarized  below: 

NO  RATING  OR  SCORING 

-  Narrative 

-  Tabular 

-  Graphical 

QUALITATIVE  RATING 

-  Adjectival 

-  Color 

NUMERICAL  SCORING 

-  No  weighting 

-  Weighting  cost  sub-criteria  only 

-  Weighting  all  evaluation  criteria 
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As  indicated  above,  numerical  scores  can  be  weighted  in 
three  ways.  First  is  no  weighting  (that  is  no  numerical  weight¬ 
ing);  the  scores  provide  an  understanding  of  how  the  offerors 
rank  for  a  particular  evaluation  sub-criterion.  Second,  the 
cost  sub-criteria  alone  can  be  weighted  which  provides  an  under¬ 
standing  of  the  ranking  for  the  combined  cost  sub-criteria. 
Third,  cost  can  be  weighted  with  all  of  the  other  evaluation 
criteria.  It  is  this  latter  weighting  that  is  generally  implied 
by  numerical  scoring. 

Whether  qualitative  ratings  or  numerical  scores  are  used, 
they  should  be  considered  as  means  of  communication  with  the  SSA 
which  are  used  by  the  SSA  along  with  other  factors,  such  as 
narrative  information. 

While  numerical  scoring  of  both  Government  estimated  cost 
and  cost  realism  for  each  contractor  appears  to  have  consider¬ 
able  merit,  it  is  presently  discouraged  by  some  Defense  organi¬ 
zations.  The  advantage  of  scoring  cost  numerically,  assuming  an 
appropriate  methodology  is  used,  is  that  it  provides  the  SSA 
with  a  clearer  understanding  of  each  offeror's  relative  ranking 
than  an  adjectival  or  color  ranking.  However,  the  SSA  needs 
sufficient  flexibility  and  discretion  with  any  evaluation  method 
to  award  to  the  offeror  who  provides  the  best  value,  all  factors 
considered.  Such  flexibility  is  particularly  important  when 
using  numerical  scoring,  because  the  SSA  does  not  want  to  be 
driven  to  an  irrational  decision  based  on  an  overly  elaborate 
and  too  inflexible  scoring  system.  Careful  selection  of  scoring 
rules  and  evaluation  criteria  weights  can  minimize  any  problems. 
Also,  it  should  be  clearly  recognized  that  numerical  scores  are 
not  precise  despite  the  aura  of  precision  they  give.  Numerical 
scores  are  only  approximate  and  small  differences  in  scores  may 
not  be  significant.  Numerical  scores  should  not  be  the  sole 
basis  for  an  award.  The  SSA  must  take  into  account  all 
pertinent  quantitative  and  qualitative  information. 

The  various  evaluation  alternatives  are  discussed  in  more 
detail  in  the  following  subsections. 
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B.  NON-NUMERICAL  EVALUATION 

Narrative  evaluations  may  be  used  alone,  but  are  generally 
used  as  supplementary  information  to  the  qualitative  ratings  or 
numerical  scores.  The  narrative  should  provide  a  clear,  concise 
summary  of  the  main  strengths  and  weaknesses  of  the  proposal  in 
the  areas  defined  by  the  evaluation  criteria  (or  sub-criteria). 
Where  qualitative  ratings  are  used,  the  narrative  should  also 
provide  specific  rationale  for  the  adjective  or  color  rating. 
Where  numerical  scores  are  used,  the  narrative  should  provide 
information  that  may  not  be  reflected  in  the  scores,  but  which 
should  be  considered  in  interpreting  the  scores. 

Tabular  and  graphical  displays  may  also  be  used  alone  or  as 
supplementary  information  to  the  qualitative  ratings  or  numeri¬ 
cal  scores.  Tabular  and  graphical  displays  using  only  the  key 
cost  data  —  the  offers  and  the  Government  estimated  costs  — can 
convey  a  great  deal  of  information  if  properly  constructed.  A 
suggested  tabular  display  is  shown  below: 


CONTRACTOR 

OFFER 

GOV'T 

GOV'T  EST 

REALISM 

(C) 

ESTIMATE 

DELTA  IN  % 

DELTA  IN  % 

(G) 

(D) 

(D* ) 

A 

B 

C 

D 

> 

The  Government  estimate  delta  is  the  difference  between  the 
Government  estimate  and  the  lowest  Government  estimate  for  any 
contractor  expressed  in  percentage  terms.  The  cost  realism 
delta  is  the  difference  betweeen  the  offer  and  the  Government 
estimate  expressed  in  percentage  terms.  These  deltas  provide  a 
clear,  convenient  display  of  the  cost  premium  that  would  have  to 
be  paid  and  the  degree  of  realism  of  eac!  contractor.  The 
deltas  are  figured  as  follows: 
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Gov't  Estimated  Cost  Delta 


x  100  %  =  D 


G* 

C  —  G 

Cost  Realism  Delta  =  -  x  100  %  =  D* 

G 

Where:  C  =  Offer 

G  =  Government  Estimated  Cost 
G*  =  Lowest  Government  Estimated  Cost 
D  =  Government  Estimated  Cost  Delta  (%) 
and  D'  =  Cost  Realism  Delta  (%) 

While  the  above  tabular  display  is  simple  and  may  appear 
trivial,  it  can  be  of  great  value  to  the  source  selection  team 
and  the  SSA. 

Graphical  presentations  of  key  cost  data  may  also  be 
useful.  Two  types  are  shown  in  Figure  8.1. 

Qualitative  rating  involves  assigning  an  adjectival  or 
color  rating  or  grade  for  each  evaluation  criterion  for  each 
offeror.  The  ratings  to  be  used  are  determined  in  advance,  and 
each  possible  rating  is  clearly  defined  and  evaluation  criteria 
or  standards  are  established,  so  that  evaluators  will  be  able  to 
assign  a  rating  with  a  minimum  of  ambiguity.  Some  of  the 
adjectival  ratings  that  are  used  are:  Excellent,  Very  Good, 
Average,  Poor,  and  Unsatisfactory;  Exceptional,  Acceptable,  and 
Unacceptable;  and  A,  B,  C,  D,  and  F.  The  color  ratings 
developed  by  the  Air  Force  are:  Blue,  Green,  Yelow  and  Red, 
which  correspond  to  exceptional,  meets  standards,  fails  to  meet 
standards  and  unacceptable,  respectively. 

Adjectival  and  color  ratings  are  sometimes  converted  to 
numerical  scores  by  assigning  a  number  or  a  range  of  numbers  to 
each  adjective  or  color.  By  quantifying  the  rating,  weighting 
of  the  evaluation  criteria  to  develop  an  overall  rating  is 
facilitated.  The  alternative  is  a  very  subjective  weighting. 

One  of  the  main  disadvantages  of  adjectival  and  color 
ratings  is  that  two  offerors  may  be  much  closer  in  relative 
merit  than  their  ratings  indicate.  This  is  because  the  ratings 
represent  a  s t e p - f unc t io n ,  and  offerors  on  the  borderline 
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between  two  ratings  may  be  very  close  but  be  given  different 
ratings . 

C.  NUMERICAL  SCORING 

The  basic  notion  for  determining  a  numerical  score  is  very 
straightforward:  for  the  Government  estimated  cost  (G),  the 
greater  the  Government  estimate  the  lower  the  score;  for  cost 
realism,  the  greater  the  difference  between  the  Government 
estimate  (G)  and  the  offer  (C),  the  lower  the  score.  This  is 
shown  graphically  below: 


GOVERNMENT  ESTIMATE 
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These  figures  are  general  representations  of  scoring  rules. 
"A”,  "B",  and  "B"'  are  arbitrary  points;  "G"  is  the  Government 
estimated  cost  for  a  particular  contractor  and  "C"  is  the 
contractor's  offer.  The  change  in  score  as  a  result  of  moving 
along  the  line  AB'  for  Government  estimated  cost  and  along  the 
line  AB  or  AB'  for  cost  realism  is  shown  as  a  straight  line  for 
simplicity,  but  in  the  general  case  the  lines  might  be  some 
other  shape,  as  discussed  below.  The  score  is  usually  based  on 
a  scale  of  0  to  10  with  10  being  the  maximum  score. 

The  scoring  rules  are  discussed  in  detail  below.  The 
Government  estimated  cost  scoring  rules  are  discussed  first. 
The  cost  realism  scoring  rules  are  discussed  beginning  on  page 
8-13.  Design-to-Unit  Production  Cost  (DTUPC)  and  Life  Cycle  Cost 
(LCC)  scoring  rules  are  not  discussed  in  this  Handbook,  but  the 
same  procedures  discussed  below  for  Government  estimated  cost 
could  be  used. 


COST  REALISM 
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Government  Estimated  Cost  Scoring  Rule 

As  mentioned  above,  the  basic  notion  for  scoring  Government 
estimated  cost  is  very  straightforward:  the  greater  the  Govern¬ 
ment  estimate,  the  lower  the  score.  Referring  to  the  above 
figure  for  Government  estimated  cost,  the  score  associated  with 
any  Government  estimated  cost,  G  (where  G  differs  for  each 
offeror),  depends  on  the  shape  of  the  line  AB*  and  the  values 
for  B  and  B*.  Some  of  the  shapes  the  scoring  rule  could  take 
are  triangle,  trapezoid,  stepped  and  various  smooth  curves 
(e.g.,  a  normal  curve),  as  represented  below: 

t 

LU 

a: 
o 
o 


The  triangle  is  a  simple  scaling  rule.  For  a  Government 
estimated  cost  of  B,  the  score  is  a  maximum  (e.g.,  10).  For 
Government  estimates  greater  than  B,  the  score  decreases 
lineraly  to  a  minimum  (e.g.,  0)  at  B*.  B  may  be  less  than  or 
equal  to  the  lowest  Government  estimate,  G*,  for  the  different 
offerors  (if  B  is  greater  than  G*,  then  in  effect  the  triangle 
is  a  trapezoid  scoring  rule). 

The  trapezoid  scoring  rule  differs  from  the  triangle 
scoring  rule  only  in  that  the  top  of  the  triangle  is  cut  off, 
such  that  Government  estimates  sufficiently  close  to  the  lowest 
Government  estimate  (e.g.,  within  a  couple  of  percent)  receive 
the  maximum  score. 

The  stepped  scoring  rule  is  another  variation  of  the 
triangle  and  derives  from  assigning  the  same  score  to  a  range  of 
costs.  For  example,  a  score  of  10  is  assigned  to  all  contrac¬ 
tors  with  Government  estimates  within  5  percent  of  B,  a  score  of 
9  to  all  contractors  within  a  range  5  to  10  percent  of  B,  etc. 
While  the  stepped  scoring  rule  can  be  presented  in  a  simple 
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table,  it  has  the  undesirable  property  that  it  may  give  quite 
different  scores  to  two  contractors  with  similar  Government 
estimates.  For  instance,  based  on  the  above  example,  two  con¬ 
tractors  with  Government  estimates  that  are  4.9  and  5.1  percent 
higher  than  B  would  receive  scores  of  10  and  9,  respectively. 
As  mentioned  earlier,  adjectival  and  color  ratings  are  also 
step-functions. 

Various  smooth  curves  can  also  be  used  as  scoring  rules.  A 
normal  curve  is  a  particularly  convenient  form  of  a  smooth 
curve.  The  parameters  of  a  normal  curve  can  be  chosen  such  that 
a  normal  curve  scoring  rule  closely  approximates  a  triangle  or 
trapezoid  scoring  rule.  The  advantage  of  a  normal  curve  scoring 
rule  is  that  it  does  not  have  a  sharp  discontinuity  at  B*  as  do 
the  the  triangle  and  trapezoid. 

A  triangle  scoring  rule  or  a  normal  curve  scoring  rule 
approximating  a  triangle  are  the  preferred  shapes  for  a  Govern¬ 
ment  estimated  cost  scoring  rule.  The  main  reason  is  that  the 
score  decreases  linearly  as  the  Government  estimated  cost 
increases,  which  appears  sensible.  Also,  these  scoring  rules 
give  more  logical  results  than  a  trapezoid  scoring  rule  for  the 
tradeoff  analyses  discussed  in  Section  IX.  A  stepped  scoring 
rule  is  not  preferred  for  the  reasons  already  given. 

While  the  normal  curve  scoring  rule  has  the  advantage  over 
the  triangle  scoring  rule  of  being  a  smooth,  continuous  curve, 
it  is  not  as  easy  to  explain  and  understand.  Furthermore,  a 
normal  curve  is  generally  associated  with  statistical  probabili¬ 
ties  or  with  uncertainties--connotations  which  are  not  approp¬ 
riate  in  this  case.  Also,  the  scores  given  by  the  two  types  of 
scoring  rules  can  be  very  similar  (depending  on  the  curve  param¬ 
eters  chosen)  in  the  range  B  to  B'.  Therefore,  the  following 
discussion  assumes  a  triangle  scoring  rule.  Normal  curve  scoring 
rules  are  discussed  in  Appendix  E. 

In  order  to  use  the  triangle  scoring  rule,  four  values  must 
be  d e t e r mi n e d -- t he  maximur  score,  the  minimum  score,  B  and  B' 
(refer  to  the  figure  on  page  8-6).  It  is  convenient  to  set  the 
maximum  score  at  10.  The  minimum  score  may  be  set  at  0  or  1. 
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With  a  minimum  score  of  1,  it  is  easier  to  define  triangle  and 
normal  curve  scoring  rules  which  approximate  each  other,  as 
discussed  in  Appendix  E.  For  this  reason,  a  value  of  1  is 
preferred  in  this  Handbook.  Also,  by  using  a  value  of  1  each 
offeror  gets  a  minimum  score  of  1  for  effort.  B,  the  Government 
estimate  for  which  the  score  is  a  maximum  of  10,  should  general¬ 
ly  be  set  equal  to  the  lowest  Government  estimate  for  any 
offeror,  G*,  for  the  following  reasons.  If  G*  is  substantially 
less  than  B,  then  a  trapezoid  is  formed  with  a  maximum  score 
given  for  G*  as  well  as  for  Government  estimates  substantially 
more  than  G*.  If  G*  is  substantially  more  than  B,  then  even  the 
offeror  with  the  lowest  Government  estimate  will  have  a  score 
substantially  less  than  the  maximum.  The  effect  of  different 
values  for  B  is  shown  graphically  below: 


The  remaining  value  that  has  to  be  established  is  B',  the 
Government  estimated  cost  above  which  the  score  is  a  minimum  of 
1.  First,  it  is  convenient  to  express  the  "Cost"  on  the  X-axis 
of  the  center  figure  above  in  terms  of  the  Government  cost 
estimate  delta,  D,  defined  above  for  the  tabular  display.  The 
delta  is  the  difference  between  the  Government  estimate  and  the 
lowest  Government  estimate  for  any  contractor  expressed  in 
percentage  terms.  That  is  D  equals  (G-G*)/G*.  The  delta  for  a 
maximum  score  is  zero  [(G*-G*)/G*  =  0],  That  is  the  offeror 
with  the  lowest  Government  estimate  receives  the  maximum  score. 
The  delta  at  which  the  score  reaches  a  minimum  score  of  1  is 
termed  the  "spread"  of  the  scoring  rule  [  (  B  ’  -  G  *  )  /  G  *  =  S  ] .  For 
example,  if  the  spread  is  40  percent,  an  offeror  whose  Govern- 
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ment  estimated  cost  is  40  percent  greater  than  the  lowest 
Government  estimated  cost  for  any  of  the  competitors  receives  a 
score  of  1.  An  offeror  whose  Government  estimated  cost  is  20 
percent  greater  receives  a  score  of  5.5.  By  using  the  delta 
instead  of  the  Government  estimated  cost  as  the  X-axis,  the 
magnitude  of  the  difference  between  the  offerors  is  readily 
apparent,  and  all  offerors  can  be  sho-.rn  on  a  single  graph  of 
the  scoring  rule.  The  triangle  scoring  rule  is  shown  below 
using  the  delta--both  the  general  case  and  an  example  with  the 
spread  equal  to  40  percent  are  shown. 


Having  established  the  maximum  and  minimum  scores  and 
having  set  B  equal  to  G*,  the  scoring  rule  is  a  function  only  of 
the  spread,  S.  Since  all  offerors  with  D  greater  than  S  will 
have  the  same  minimum  score  of  one,  if  S  is  too  small,  there 
will  not  be  sufficient  differentiation  between  offerors.  And, 
if  S  is  too  large,  even  very  expensive  offerors  (as  measured  by 
the  Government  estimates)  will  still  receive  scores  above  the 
minimum  of  one  and  large  differences  in  the  Government  estimates 
will  result  in  only  small  differences  in  score.  Therefore,  the 
spread  must  be  chosen  with  care.  It  is  important  for  the  source 
selection  team  to  analyze  the  effects  of  different  spreads — 
different  values  of  S — as  well  as  different  weights  for  the 
Government  estimated  cost  sub-criterion.  The  determination  of 
the  spread  is  discussed  next.  The  determination  of  the  cost 
evaluation  weights  will  be  discussed  later  in  this  section. 
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Government  Estimated  Cost  Scoring  Rule  Spread 

As  mentioned  above,  if  the  spread  is  too  small,  there  will 
not  be  sufficient  dif f erentation  between  offerors.  And,  if  the 
spread  is  too  large,  even  very  expensive  offerors  (as  measured 
by  the  Government  estimates)  will  still  receive  scores  well 
above  the  minimum.  This  is  illustrated  below: 

SCORES  FOR  DIFFERENT  SPREADS  AND  DELTAS 
,  Delta  (D) 
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At  what  point  should  the  Government  estimate  for  an  offeror 
be  considered  too  high  relative  to  the  Government  estimates  for 
other  offerors?  Given  that  all  offerors  propose  to  the  same  set 
of  requirements,  their  costs  should  not  differ  widely  except 
possibly  to  account  for  such  factors  as  variant  design 
approaches,  labor  and  overhead  rates,  and  efficiencies.  It 
seems  reasonable  that  there  would  be  a  much  greater  likelihood 
that  G  would  be  reasonably  close  to  G*,  say  within  10  to  20 
percent,  than  far  from  G*,  say  50  to  100  percent. 

To  test  this.  Government  estimates  for  eight  NAVELEX  pro¬ 
grams  were  examined.  The  Government  estimate  or  estimates  which 
were  not  the  lowest  for  each  program  are  shown  in  the  histogram 
in  Figure  8-2  (BAF0  estimates  were  used).  When  a  normal  curve 
is  fitted  to  the  histogram,  the  resulting  standard  deviation  (a) 
is  18  percent.  In  other  words,  there  was  a  68  percent  likeli¬ 
hood  that  the  Government  estimates  were  within  18  percent  of  the 
lowest  Government  estimate  and  a  95  percent  likelihood  that  the 
Government  estimates  were  within  36  percent  (2a).  The  (2a) 
point  corresponds  roughly  to  the  spread,  S,  of  a  triangle 
scoring  rule  (see  Appendix  E  for  an  explanation).  Based  on  this 
sample,  a  scoring  rule  with  a  spread  of  30  to  40  percent  would 
provide  a  reasonable  differentiation  between  offerors. 
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HISTOGRAM:  RELATIVE  FREQUENCY  OF  OCCURRENCE 


NORMAL  CURVE:  PROBABILITY  DENSITY 


The  NAVELEX  data  were  also  analyzed  as  follows.  The 
Government  estimate  or  estimates  which  were  not  the  lowest  were 
16  percent  greater  on  average  than  the  G*  for  each  program. 
Half  of  the  time  the  Government  estimate  that  was  not  the  lowest 
was  within  10  percent  of  G  *  and  two  thirds  of  the  time  was 
within  20  percent.  The  greatest  difference  from  G*  was  a  little 
more  than  40  percent.  Again,  based  on  the  sample,  a  spread  of 
30  to  40  percent  seems  reasonable. 

There  is  no  "best"  spread  to  use.  Different  spreads  may  be 
appropriate  for  different  stages  of  development,  different  types 
of  hardware,  and  other  program  differences.  Therefore,  in 
determining  an  appropriate  spread,  the  source  selection  cost 
evaluation  team  should  consider  these  various  factors,  as  well 
as  how  much  it  wants  to  differentiate  scores  for  offerors  with 
various  deltas.  And,  sensitivity  and  tradeoff  analyses  of  the 
effects  of  different  scoring  rule  spreads  and  different  weights 
should  be  made,  as  discussed  in  Section  IX. 

Cost  Realism  Scoring  Rule 

Earlier  in  this  section  it  was  mentioned  that  the  basic 
notion  for  scoring  cost  realism  is  very  straightforward:  the 
greater  the  difference  between  the  Government  estimate  (G)  and 
the  offer  (C),  the  lower  the  score.  The  graph  that  was  shown  is 
repeated  below: 

COST  REALISM 


COST 


AB  and  AB'  are  shown  as  straight  lines  for  simplicity,  but  in 
the  general  case  might  be  some  other  shape.  The  score  asso¬ 
ciated  with  any  difference  (C-G)  depends  on  the  shape  of  the 


line  BAB’  and  the  values  for  B  and  B'.  The  differences  B’-G  and 
G-B  may  be  the  same,  in  which  case  the  scoring  rule  is  "symmet- 
rica  1"  (assuming  the  lines  AB  and  AB'  are  the  same  shape).  If 
B'-G  and  G-B  are  different,  then  the  scoring  rule  is  "skewed.” 
There  does  not  appear  to  be  an  obvious  rationale  for  giving  one 
offeror  at  some  percentage  below  the  Government  estimate  a 
different  score  from  another  the  same  percentage  above  the 
Government  estimate.  For  this  reason  and  for  simplicity, 
symmetry  is  assumed  in  the  following  discussion.  Some  of  the 
shapes  the  scoring  rule  could  take  are  triangle,  trapezoid, 
stepped,  and  various  smooth  curves  (e.g.,  a  normal  curve). 
These  shapes  were  illustrated  earlier  for  the  Government  esti¬ 
mated  cost,  and  the  illustrations  are  repeated  below: 


COST—  COST—  COST—  COST— 


The  triangle  is  a  simple  scaling  rule.  If  the  difference 
between  G  and  C  is  zero,  the  score  is  a  maximum  —  say,  10.  If  C 
is  more  than  B'  or  less  than  B,  the  score  is  a  minimum — say, 
zero.  In  between,  the  score  is  proportional  to  the  difference 
between  G  and  C. 

The  trapezoid  scoring  rule  differs  from  the  triangle 
scoring  rule  only  in  that  the  top  of  the  triangle  is  cut  off, 
giving  it  a  trapezoid  or  mesa  shape.  The  rationale  is  that 
offers  sufficiently  close  to  the  Government  estimate  (e.g., 
within  a  couple  of  percent)  should  receive  the  maximum  score. 
This  gives  some  recognition  of  the  uncertainty  associated  with 
the  Government  estimate.  Uncertainty  is  discussed  later. 

The  stepped  scoring  rule  is  another  variation  of  the 
triangle  and  derives  from  assigning  the  same  score  to  a  range  of 
offers.  As  discussed  earlier,  the  stepped  scoring  rule  has  the 
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undesirable  property  of  attributing  quite  different  scores  to 
two  contractors  with  similar  degrees  of  cost  realism  (i.e.,  if 
the  two  scores  are  on  either  side  of  a  step  break). 

Various  smooth  curves  can  also  be  used  as  scoring  rules.  A 
normal  curve  (a  "bell”  shaped  curve)  is  a  particularly 
convenient  form  of  a  smooth  curve.  The  parameters  of  a  normal 
curve  can  be  chosen  such  that  a  normal  curve  scoring  rule  close¬ 
ly  approximates  a  triangle  or  trapezoid  scoring  rule.  The 
advantage  of  a  normal  curve  scoring  rule  is  that  it  does  not 
have  sharp  discontinuities  at  B  and  B'  as  do  the  triangle  and 
trapezoid . 

A  trapezoid  scoring  rule  or  a  normal  curve  scoring  rule  are 
the  preferred  shapes  for  a  cost  realism  scoring  rule.  The  main 
reason  is  that  offers  sufficiently  close  to  the  Government  esti¬ 
mate  (e.g.,  within  a  couple  of  percent)  receive  the  maximum 
score  or  very  close  to  the  maximum  score.  As  stated  above,  this 
gives  some  recognition  of  the  uncertainty  associated  with  the 
Government  estimate.  A  stepped  scoring  rule  is  not  preferred 
for  the  reasons  already  given. 

The  following  discussion  assumes  a  trapezoid  scoring  rule. 
Normal  curve  scoring  rules  are  discussed  in  Appendix  E. 

In  order  to  use  the  trapezoid  scoring  rule,  six  values  must 
be  determined — the  maximum  score,  the  minimum  score,  B,  B',  and 
the  range  of  values  for  which  the  score  is  a  maximum  —  the 
"plateau".  The  scoring  rule  is  shown  below. 
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It  is  convenient  to  set  the  maximum  score  at  10.  The 
minimum  score  may  be  set  at  0  or  1.  A  value  of  1  is  preferred 
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for  the  same  reasons  given  for  the  Government  estimated  cost 
scoring  rule. 

Before  proceeding,  it  is  convenient  to  express  the  "Cost” 
on  the  X-axis  of  the  scoring  rule  in  terms  of  the  cost  realism 
delta,  D',  defined  earlier  for  the  tabular  display.  The  delta 
is  the  difference  between  the  offer  and  the  Government  estimate 
expressed  in  percentage  terms.  That  is  D*  equals  (C-G)/G.  It 
is  also  convenient  to  assume  symmetry  such  that  offers  higher  or 
lower  than  the  Government  estimate  by  the  same  amount  receive 
the  same  score.  If  the  Government  estimate  is  the  same  as  the 
offer,  the  score  is  a  maximum  of  10  and  the  delta  is  zero  [(C- 
G)/G  =  0].  The  score  reaches  a  minimum  of  1  for  (B-G)/G  =  -S 
for  offers  less  than  G  and  for  (B'-G)/G  =  S  for  offers  greater 
than  G.  S  is  termed  the  "spread"  and  has  the  same  meaning  as 
for  the  Government  estimated  cost  scoring  rule.  The  plateau, 
the  deltas  for  which  the  score  is  a  maximum  of  10,  should  be 
small  compared  to  the  total  spread  of  -S  to  S.  It  is  convenient 
to  set  the  plateau  at  10  percent  of  the  total  spread.  Thus,  the 
score  is  a  maximum  of  10  from  -0.1  S  to  0.1  S.  The  trapezoid 
scoring  rule  is  shown  below  using  the  delta--both  the  general 
case  and  an  example  with  the  spread  equal  to  20%  are  shown. 
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COST  RELATIVE  TO  G  {  '.) 


As  indicated  in  the  figure  on  the  right,  an  offeror  whose  offer 
is  20  percent  higher  or  lower  than  the  Government  estimate 
receives  a  score  of  1.  An  offeror  whose  offer  is  10  percent 
higher  or  lower  receives  a  score  of  6.  An  offeror  within  two 
percent  of  the  Government  estimate  receives  a  score  of  10.  By 
using  the  delta  instead  of  the  cost  as  the  X-axis,  the  degree  of 
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cost  realism  is  readily  apparent,  and  all  ->fferors  can  be  shown 
on  a  single  graph  of  the  scoring  rule. 

Having  established  the  maximum  and  minimum  scores  and 
having  set  the  plateau  at  some  fixed  percentage  of  the  spread 
(e.g.,  10  perecent),  the  scoring  rule  is  a  function  only  of  the 
spread,  S.  Since  all  offerors  with  D*  greater  than  S  will  have 
the  same  minimum  score  of  one,  if  S  is  too  small,  there  will  not 
be  sufficient  differentiation  between  offerors.  And,  if  S  is 
too  large,  even  very  unrealistic  offerors  will  still  receive 
scores  above  the  minimum  of  one  and  large  differences  in  the 
degree  of  realism  will  result  in  only  small  differences  in 
score.  Therefore,  the  spread  must  be  chosen  with  care.  It  is 
important  for  the  source  selection  team  to  analyze  the  effects 
of  different  spreads  —  different  values  of  S  —  as  well  as 
different  weights  for  the  cost  realism  sub-criterion.  The  deter¬ 
mination  of  the  spread  is  discussed  next.  The  determination  of 
the  cost  evaluation  weights  will  be  discussed  later  in  this 
section . 

Cost  Realism  Scoring  Rule  Spread 

As  mentioned  above,  if  the  spread  is  too  small,  there  will 
not  be  sufficient  differentiation  between  offers.  And,  if  the 
spread  is  too  large,  even  very  unrealistic  offers  will  still 
receive  scores  well  above  the  minimum  score  and  large  differen¬ 
ces  in  realism  will  result  in  only  small  differences  in  score. 

A  spread  of  15  to  25  percent  appears  to  be  reasonable  for 
defining  a  cost  realism  scoring  rule.  Offers  more  than  15  to  20 
percent  from  a  carefully  prepared  Government  estimate,  relying 
primarily  on  the  offeror's  own  data,  certainly  raise  a  question 
as  to  cost  realism.  Of  course,  what  is  realistic  will  depend  to 
some  extent  on  the  nature  of  the  development  program--the  risks 
involved,  how  much  of  the  system  has  been  previously  designed, 
etc.  It  seems  reasonable  to  assume  lat  offers  more  than  15  to 
25  percent  from  the  Government  estimate,  depending  upon  the 
particular  program,  are  unrealistic  and  should  receive  close  to 
a  minimum  score.  The  low  end  of  this  suggested  range,  15 
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percent,  would  apply  only  if  the  program  was  well  defined,  there 
were  few  risks  or  unknowns  and  there  existed  a  good  data  base  on 
which  to  base  cost  estimates. 

A  range  of  no  more  than  25  percent  for  the  scoring  rule 
spread  is  supported  by  data  for  eight  NAVELEX  programs.  Cost 
realism  (i.e.,  D')  was  determined  for  19  offers  made  on  the 
eight  programs,  as  shown  in  the  histogram  in  Figure  8-3  (BAFO 
estimates  were  used).  When  a  normal  curve  is  fitted  to  the 
histogram,  the  resulting  standard  deviation  (o)  is  13  percent. 
In  other  words,  there  was  a  68  percent  likelihood  that  the 
offers  were  within  13  percent  of  the  Government  estimate  and  a 
95  percent  likelihood  that  the  offers  were  within  27  percent 
(2a).  The  2a  point  corresponds  roughly  to  the  spread,  S,  of  a 
trapezoid  scoring  rule  (see  Appendix  E  for  an  explanation). 
Based  on  this  sample,  a  scoring  rule  with  a  spread  of  25  percent 


or  less  would  provide  a  reasonable  differentiation  between 
offerors  . 

The  NAVELEX  data  were  also  analyzed  as  follows.  The 
average  cost  xeaiism  delta,  D',  was  9  percent,  and  84  percent  of 
the  time  D1  was  less  than  25  percent.  Again,  based  on  the 
sample,  a  spread  of  25  percent  or  less  seems  reasonable. 

There  is  no  "best"  spread  to  use.  Different  spreads  may  be 
appropriate  for  different  stages  of  development,  different  types 
of  hardware,  and  other  program  differences.  Therefore,  in 
determining  an  appropriate  spread,  the  source  selection  cost 
evaluation  team  should  consider  these  various  factors,  as  well 
as  how  much  it  wants  to  differentiate  scores  for  offerors  with 
various  deltas.  And,  sensitivity  and  tradeoff  analyses  of  the 
effects  of  different  scoring  rule  spreads  and  different  weights 
should  be  made,  as  discussed  in  Section  IX.  Consideration 
should  also  be  given  to  the  potential  uncertainties  in  the 
Government  estimates,  and  the  scoring  rule  spread  should  be  at 
least  as  great  at  the  uncertainty,  as  discussed  next. 

Uncertainty 

The  previous  section,  on  the  Government  estimate,  briefly 
discussed  uncertainty,  pointing  out  that  considerable  uncertain- 
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ty  in  the  Government  estimate  derives  from  technical  and  program 
(e.g.,  schedule)  risks  and  from  uncertainties  associated  with 
cost  estimating  data  and  techniques.  It  is  therefore  important 
to  take  into  account  the  uncertainty  associated  with  the  Govern¬ 
ment  estimate  when  judging  the  realism  of  the  offeror's  proposed 
cost.  The  greater  the  degree  of  uncertainty  in  the  Government 
estimate,  the  more  admissible  are  greater  differences  between  an 
offer  made  by  a  contractor  and  the  Government  estimated  cost  for 
that  contractor. 

A  cost  realism  scoring  rule  can  make  two  kinds  of  errors, 
as  pointed  out  by  Ling  and  Wal lenius*: 

Type  A  error--assigning  a  higher  score  than  warranted  to  an 
improbable  proposed  cost,  and 

Type  B  error — assigning  a  lower  score  than  deserved  to  a  highly 
plausible  proposed  cost. 

Type  B  errors  result  if  the  spread  in  uncertainty  is  greater 
than  the  spread  in  the  scoring  rule.  This  is  undesirable 
because  offers  with  a  reasonable  likelihood  of  matching  the 
Government  estimate  would  receive  a  score  at  or  close  to  the 
minimum.  This  is  illustrated  below  with  a  trapezoid  scoring 
rule  and  a  hypothetical  uncertainty  distribution: 


*  Ling  and  Wallenius,  Cost  Realism,  Technical  Report  #400, 
Department  of  Mathematical  Sciences,  Clemson  University, 
December  31,  1982. 


Offeror  A  receives  the  minimum  score  even  though  there  is  a 
reasonable  likelihood  it  could  in  fact  match  or  closely  approxi¬ 
mate  the  Government  estimate. 

Type  A  errors  result  if  the  spread  in  uncertainty  is  less 
than  the  spread  in  the  scoring  rule.  This  does  not  appear  to  be 
of  concern  as  far  as  scoring  is  concerned.  Although  various 
offers  may  be  u nr ea 1 i s t i c--c 1 ea r 1 y  more  or  less  than  they  would 
have  been  had  the  offerors  been  realistic  or  more  careful  —  the 
scoring  rule  should  differentiate  among  these  unrealistic  esti¬ 
mates,  at  least  to  a  point.  However,  the  scoring  rule,  if  given 
the  same  spread  as  the  uncertainty,  would  give  a  minimum  cost 
realism  score  to  all  unrealistic  offers  with  deltas  (D*)  greater 
than  the  spread  (S).  This  is  illustrated  below: 


For  example,  if  various  offers  had  deltas  between  zero  and  20 
percent  and  the  spread  was  significantly  less  than  20  percent, 
say  10  percent,  then  all  offers  with  deltas  between  10  and  20 
percent  would  receive  the  minimum  score,  and  thus  there  would  be 
no  differentiation  of  scores. 

It  seems  unlikely  that  the  uncertainty  spread  would  be 
greater  that  a  scoring  rule  spread  of  15  to  25  percent  given 
that  the  uncertainty  and  risk  associated  with  a  Government 
estimate  is  a  function  of  the  stated  requirements  in  the  solici¬ 
tation  (e.g.,  no  scope  changes  or  redirections)  and  that  the 
main  uncertainty  will  be  in  the  interpretation  and  use  of  each 
offeror’s  cost  data  and  history.  Also,  since  the  offers  and 
Government  estimates  generally  use  audited  labor  and  overhead 
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rates,  no  uncertainty  in  these  rates  is  generally  taken  into 
account. 

One  method  for  understanding  the  possible  effects  of  uncer¬ 
tainty  in  the  Government  estimate  on  the  cost  realism  scores  is 
to  perform  a  sensitivity  analysis  to  see  the  effects  of  a  few 
percent  change  in  the  Government  estimate  on  the  cost  realism 
scores.  Any  change  in  the  ranking  of  two  offerors,  due  to  such 
a  small  change  in  the  Government  estimate,  would  indicate  that 
there  is  no  significant  difference  in  the  cost  realism  of  these 
offerors . 

In  conclusion,  unless  there  is  concern  that  the  uncertainty 
spread  is  greater  than  the  scoring  rule  spread,  it  does  not 
appear  necessary  to  determine  uncertainty  in  the  Government 
estimate  for  a  particular  program  or  to  adjust  the  spread  of  the 
scoring  rule  accordingly.  Benefits  other  than  to  cost  realism 
scoring  may,  however,  derive  from  determining  the  uncertainty. 
An  uncertainty  assessment  would  clearly  lead  to  a  better  under¬ 
standing  of  the  cost  estimates  and  would  highlight  certain  areas 
warranting  closer  attention  in  the  source  selection  evaluation 
and  closer  control  during  contract  execution. 

Combined  Scores 

The  overall  cost  evaluation  score  depends  on  the  scores  for 
Government  estimated  cost  and  cost  realism.  The  overall  source 
selection  evaluation  score  depends  upon  the  scores  and  the 
weights  for  each  factor — cost,  technical,  management,  etc.  The 
cost  evaluation  team  with  assistance  from  the  technical  team,  as 
appropriate,  should  analyze  the  effects  of  (1)  different  scoring 
rules  (e.g.,  trapezoid  versus  normal  curve),  (2)  different  scor¬ 
ing  rule  spreads,  and  (3)  different  weights.  The  trapezoid  and 
normal  curve  scoring  rules  can  produce  very  similar  results  as 
dicussed  above.  The  triangle  and  trapezoid  scoring  rules  have 
been  emphasized  because  they  are  easier  to  explain  and  under¬ 
stand.  However,  some  may  prefer  the  normal  curve  which  is 
discussed  in  detail  in  Appendix  E.  The  determination  of  appro¬ 
priate  scoring  rule  spreads  based  ^  imarily  on  differentiating 
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the  scores  of  various  offerors  was  discussed  above.  The  deter¬ 
mination  of  appropriate  spreads  and  weights  through  sensitivity 
analyses  and  tradeoff  analyses  is  the  subject  of  the  next 
section . 

Miscellaneous  Considerations 

Some  miscellaneous  considerations  related  to  numerical 
scoring  are  as  follows: 

a.)  This  Handbook  assesses  cost  realism  in  terms  of  the 
difference  between  an  offer  and  the  Government  estimate  of  the 
offeror’s  total  cost.  Cost  realism  could  also  be  assessed  in 
terms  of  the  sum  of  the  absolute  values  of  the  differences  for 
each  WBS  item.  Differences  at  the  WBS  item  level  are  of  some 
interest.  However,  generally  the  main  concern  in  assessing  cost 
realism  is  the  comparison  of  the  total  offer  to  the  total 
Government  estimate  in  order  to  indicate  whether  an  offeror  may 
be  buying-in,  which  would  put  the  Government  into  a  built-in 
cost  growth  position.  Furthermore,  total  cost  estimates  are 
better  than  they  would  be  otherwise,  given  the  high  degree  of 
uncertainty  in  cost  estimates,  because  errors  on  the  high  side 
and  errors  on  the  low  side  cancel  out  to  some  extent.  Likewise, 
in  execution  of  a  contract,  underspending  in  some  WBS  line  items 
and  overspending  in  others  will  cancel  to  some  extent.  There¬ 
fore,  it  seems  fair  to  assess  cost  realism  in  terms  of  the 
algebraic  sum  of  the  differences  between  the  offer  and  the 
Government  estimate  at  the  WBS  item  level  (i.e.,  the  net 
differences)  and  not  the  sum  of  the  absolute  values  of  the 
differences  at  the  WBS  level.  But,  as  stated  above,  differences 
at  the  WBS  item  level  are  of  interest.  For  example,  there  would 
be  more  confidence  in  an  offeror  whose  small  differences  at  the 
WBS  item  level  completely  cancelled  out  than  in  an  offeror  with 
large  differences  which  also  cancelled  out  completely.  Large 
differences  may  indicate  that  an  offeror  lacks  understanding  of 
the  requirements  and  should  be  highlighted  in  the  narrative  cost 
evaluation . 
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b. )  For  a  particular  procurement,  the  shape  and  spread  of 
the  uncertainty  distribution  for  the  Government  estimate  would 
differ  for  each  of  the  offerors.  Because  of  the  difficulty  in 
characterizing  uncertainty,  it  is  difficult  to  know  both  the 
extent  of  such  differences  and  their  significance.  In  general, 
however,  these  differences  should  not  be  significant.  Each 
offeror  proposes  to  accomplish  the  same  objectives  with  general¬ 
ly  similar  designs,  risks  and  the  like.  However,  there  could  be 
instances  of  significant  difference  in  approach  or  in  other 
factors  which  could  have  significantly  different  Government  cost 
uncertainties.  For  example,  the  Government  may  want  a  helicop¬ 
ter  exhibiting  certain  characteristics;  one  offeror  may  propose 
an  essentially  new  helicopter  and  another  offeror  may  propose  a 
modified  version  of  a  model  he  already  produces.  Presumably, 
the  Government  estimate  would  be  less  uncertain  for  the  latter 
offeror.  Application  of  the  same  scoring  rule  to  both  offers 
could  result  in  identical  scores,  even  though  the  likelihood 
that  their  offers  would  match  the  Government  estimate  differs 
greatly.  The  cost  evaluation  team  should  be  aware  of  these 
possibilities  and  try  to  assure  that  the  scoring  process  does 
not  penalize  offerors  unjustly.  If  it  can  be  anticipated  that 
proposals  with  approaches  subject  to  very  different  cost  uncer¬ 
tainties  will  be  received,  then  a  cost  evaluation  approach  can 
be  developed  and  prospective  offerors  notified  in  the  solic¬ 
itation  that  the  evaluation  methods  will  differ  depending  upon 
certain  factors  (e.g.,  new  or  modified  hardware).  If  the  prob¬ 
lem  only  becomes  apparent  after  proposals  are  received,  then 
modified  evaluation  methods  could  be  announced  to  the  offerors 
before  receipt  of  their  best-and-f ina 1  offers. 

c. )  A  computer  model  has  been  developed  for  calculating 
scores  and  is  described  in  Appendix  F. 

Rationalizing  Numerical  Scoring 

As  stated  in  the  beginning  of  this  section,  numerical 
scoring  of  costs  is  discouraged  by  some  Defense  organizations. 
The  main  objection  appears  to  be  that  an  inflexible  scoring  and 
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weighting  system  will  force  an  irrational  decision  based  on  an 
inflexible  scoring  and  weighting  system. 

The  advantage  of  numerical  scoring  is  that  comparisons  are 
more  structured  and  less  arbitrary.  And,  there  are  several 
means  available  to  avoid  or  lessen  the  chances  of  being  driven 
to  an  irrational  decision.  These  include: 

a. )  Careful  selection  of  scoring  rules  and  weights  with 
the  help  of  sensitivity  and  tradeoff  analyses  (as  discussed  in 
the  next  section)  can  minimize  any  problems  and  surprises. 

b. )  The  SSA  generally  has  enough  flexibility,  if  he 
chooses  to  exercise  it,  to  make  an  award  to  the  best  offeror, 
all  things  considered,  even  if  there  is  a  numerical  scoring 
system.  Obviously,  if  the  numerical  scoring  system  appears  to 
lead  to  a  bad  decision,  the  SSA  must  be  able  to  support  any 
decision  to  disregard  it. 

c. )  If  the  numerical  scoring  system  is  driving  toward  an 
irrational  decision,  the  SSA  can  issue  a  revision  to  the  solici¬ 
tation  prior  to  the  best-and— f ina 1  offer.  The  revision  can 
state  that  the  evaluation  criteria  have  been  revised  (without, 
of  course,  revealing  the  precise  criteria  and  scoring  system). 

d. )  The  SSA  can  recompete. 

e. )  Cost  scores  can  be  developed  and  provided  to  the  SSA 
as  advisory  information  without  incorporating  them  into  an 
overall  numerical  score  (as  discussed  in  the  beginning  of  this 
section ) . 

D.  EVALUATION  METHODOLOGY  SUMMARY 

There  are  several  alternatives  for  evaluating  Government 
estimated  cost  and  cost  realism.  Narrative,  tabular  and  graphi¬ 
cal  displays  can  be  used  without  any  rating  or  scoring  or  as 
information  which  supplements  qualitative  ratings  or  numerical 
scores . 

Qualitative  ratings  may  be  used  by  assigning  an  adjective 
or  color  rating  or  grade  for  each  offeror.  Numerical  scoring 
may  be  used  by  developing  a  realistic  Government  cost  estimate 
for  each  of  the  offerors  and  using  triangle,  trapezoid  or  normal 
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curve  scoring  rules.  The  methodology  for  numerical  scoring  is 
summarized  below. 

The  Government  estimated  cost  score  is  a  function  of  the 
difference  between  the  Government  estimate  for  each  offeror  (G) 
and  the  lowest  Government  estimate  for  any  offeror  (G*) 
expressed  in  percentage  terms  as: 

/  G  -  G*  v 

Score  (GE)  =  f  ^  — — —  %  J  =  f  (D) 

The  cost  realism  score  is  a  function  of  the  difference 
between  the  offer  (C)  and  the  Government  estimate  (G)  expressed 
in  percentage  terms  as: 

C  G  \ 

Score  (CR)  =  f  (  -  %)  =  f(D') 

The  seeing  rules  are  shown  in  Figure  8.4.  For  illustra¬ 
tive  purposes,  the  maximum  scores  are  shown  as  10  and  the 
minimum  scores  as  1.  For  the  Government  estimated  cost  scoring 
rule,  a  spread  of  40  percent  is  illustrated.  For  the  cost 
realism  scoring  rule,  a  spread  of  20  percent  is  illustrated,  and 
a  "plateau"  (where  the  score  is  a  maximum)  of  10  percent  of  the 
spread  or  plus  or  minus  2  percent  of  the  Government  estimate  is 
illustrated.  Generally,  the  Government  estimated  cost  spread 
would  be  30  to  50  percent,  and  the  cost  realism  spread  would  be 
15  to  25  percent.  Normal  curve  scoring  rules  are  discussed  in 
Appendix  E. 

In  determining  values  for  the  spreads  for  any  source  selec¬ 
tion,  the  cost  evaluation  team  should  analyze  the  effects  of 
different  values  of  the  spreads  on  the  cost  score  and  the 
overall  score.  This  should  include  a  determination  of  the 
reasonableness  of  the  tradeoffs  between  cost  and  technical 
scores  and  between  Government  estimated  cost  and  cost  realism 
scores  as  discussed  in  Section  IX.  The  actual  values  establish¬ 
ed  for  the  spreads  will  of  course  be  a  subjective  determination. 

In  addition  to  providing  the  SSA  with  a  qualitative  rating 
or  numerical  score,  a  tabular  and  even  a  graphical  presentation 
of  C,  G,  (C-G)/G,  and  (G-G*)/G*  should  be  presented.  Further- 
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Figure  8.4.  Scoring  Rules 
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more,  the  SSA  should  be  provided  with  a  narrative  evaluation  of 
Government  estimated  cost  and  cost  realism. 


Methodology:  Rate  or  score  Government  estimated  cost  and 
cost  realism  for  each  contractor.  If  qualitative  rating  is 
used,  apply  a  rating  or  grade  (e.g.,  an  adjective)  for  each 
cost  evaluation  sub-criteria.  If  numerical  scoring  is  used, 
apply  a  triangle,  trapezoid  or  normal  curve  scoring  rule. 
The  Government  estimated  cost  score  is  a  function  of  the 
difference  between  the  Government  estimate  for  each  offeror 
and  the  lowest  Government  estimate  for  any  offeror.  The 
cost  realism  score  is  a  function  of  the  difference  between 
the  offer  and  the  Government  estimate.  The  differences 
corresponding  to  a  minimum  score  (triangle  or  trapezoid)  or 
to  a  95  percent  likelihood  (normal  curve)  should  be  deter¬ 
mined  for  each  procurement  based  on  sensitivity  and  tradeoff 
analyses.  Also  prepare  a  tabular,  and  possibly  graphical, 
presentation  of  the  a)  offers,  b)  Government  estimates,  c) 
percent  differences  between  the  Government  estimates  and  the 
lowest  Government  estimate,  and  d)  percent  differences 
between  the  offers  and  the  Government  estimates.  In  addi¬ 
tion,  prepare  a  narrative  cost  evaluation. 
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SECTION  IX 

SENSITIVITY  AND  TRADEOFF  ANALYSES 


It  was  stressed  in  the  numerical  scoring  discussion  in  the 
previous  section  that  cost  evaluation  scoring  rules  and  weights 
must  be  carefully  selected  in  order  to  minimize  any  problems  and 
surprises.  Selecting  the  appropriate  scoring  rule  is  primarily 
a  matter  of  determining  an  appropriate  spread.  This  section 
addresses  two  methods  for  helped  to  determine  appropriate 
scoring  rule  spreads  and  weights--sensiti vity  analyses  and 
tradeoff  analyses. 

A.  SENSITIVITY  ANALYSES 

Sensititity  analyses  involve  varying  the  spread  and/or 
weight  and  calculating  the  affect  of  these  variations  on  the 
numerical  score.  An  example  of  varying  the  Government  estimated 
cost  spread  from  30  to  50  percent  is  shown  in  Figure  9.1  for 
five  hypothetical  offerors  (A  to  E).  The  offers  and  correspond¬ 
ing  Government  estimates  for  these  hypothetical  offerors  are 
the  same  as  those  used  in  the  illustrative  example  in  Section  X. 
Sensitivity  analyses  give  some  idea  of  the  affect  of  spreads  and 
weights  on  scores  and  therefore  may  be  of  some  help  in  determin¬ 
ing  appropriate  spreads  and  weights.  A  computer  model  has  been 
developed  for  calculating  scores  and  conducting  sensitivity 
analyses  and  is  described  in  Appendix  F,  Cost  Realism  Computer 
Models.  Illustrative  sensitivity  analyses  are  provided  for  a 
broad  range  of  values  for  the  spreads  and  weights. 

Sensitivity  analyses  may  also  be  used  for  the  purpose  of 
understanding  the  effects  of  uncertainty  in  the  Government 
estimate  on  the  cost  realism  scores  as  mentioned  in  the  uncer¬ 
tainty  discussion  in  Section  VIII. 

B.  TRADEOFF  ANALYSES 

Tradeoff  analyses  involve  trading  off  technical  and  Govern¬ 
ment  oeHmated  cost,  technical  and  cost  realism,  or  Government 
estimated  cost  and  cost  realism.  For  example,  if  one  offeror 
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Figure  9.1  Sensitivity  Analysis 
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has  a  technical  score  10  percent  greater  than  that  for  another 
offeror,  how  much  lower  would  his  Government  estimated  cost 
score  have  to  be  to  offset  his  technical  advantage.  The  answer 
can  be  determined  from  the  spread  and  weight  for  the  Goverment 
estimated  cost  and  the  technical  weight.  If  the  answer,  i.e. 
the  trade,  is  not  reasonable,  then  the  scoring  rule  spread  or 
the  weights  can  be  adjusted. 

Tradeoffs  are  illustrated  in  Figure  9.2.  The  following 
weights  and  spreads  are  assumed: 


Evaluation 

Criteria 

Weight 

Spread 

Shape 

Technical 

60% 

Cost 

30% 

Gov ' t  Estimate 

(20%) 

40% 

Triangle 

Cost  Realism 

(10%) 

20% 

Trapezoid 

Other 

10% 

Total 

100% 

It  is  also  assumed  that  an  offeror  A  is  technically  superior  to 
an  offeror  B  by  10  percent — a  technical  score  of  60  for  A  and 
54.5  for  B,  out  of  a  possible  60.  For  the  Government  estimated 
cost  versus  technical  tradeoff,  offeror  A's  5.5  point  advantage 
in  technical  score  would  be  offset  if  his  Government  estimated 
cost  score  is  5.5  points  less  than  B's.  A  Government  estimated 
cost  score  of  14.5  out  of  a  possible  20  (7.3  on  a  scale  of  10) 
corresponds  to  a  Delta  ((G-G*)/G)  of  12  percent.  Offeror  B  is 
assumed  to  have  a  Delta  of  zero.  Thus,  offeror  A  would  lose  his 
advantage  in  technical  score  if  his  Government  estimated  cost 
were  12  percent  or  more  higher  than  B's,  Similarly,  for  the 
cost  realism  versus  technical  tradeoff,  offeror  A's  advantage  in 
technical  score  would  be  offset  if  his  cost  realism  score  is  5.5 
points  less  than  B's.  A  cost  realism  score  of  4.5  out  of  a 
possible  10  corresponds  to  a  Delta'  ((C-G)/G)  of  13  percent. 
Offeror  B  is  assumed  to  have  a  Delta  of  zero. 
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GOVERNMENT  ESTIMATE 
VS.  TECHNICAL 


COST  REALISM 
VS.  TECHNICAL 


MAXIMUM  OFFEROR  OFFEROR  OFFEROR 

CRITERIA  SCORE  A  B  A 

OTHER 

10 

1 

COST  REALISM 

10 

10 

10 

10 

10 

10 

4.5 

GOVERNMENT 

on 

ESTIMATE 

L.  1) 

14.5 

20 

20 

TECHNICAL 

60 

60 

54.5 

60 

TOTAL  POINTS 

100 

94.5 

94.5 

94.5 

Figure  9.2.  Cost/Technical  Tradeoffs  Example 


OFFEROR 

B 


10 


10 


20 


54.5 


94.5 
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These  tradeoffs  may  be  summarized  as  follows:  an  offeror  A 
who  is  technically  superior  to  an  offeror  B  by  10  percent  would 
receive  the  same  overall  score  as  offeror  B,  if:  (1)  offeror  A’s 
Government  estimated  cost  was  12  percent  greater  than  offeror 
B’s  Government  estimated  cost  (assumed  as  the  lowest  Government 
estimate),  or  (2)  B’s  offer  matches  the  Government  estimate  and 
A's  offer  differs  from  the  Government  estimate  by  13  percent. 
Note  that  the  cost  criterion  not  being  traded  off  is  assumed  to 
receive  the  maximum  score.  Also  note  that  there  is  a  tradeoff 
implied  between  Government  estimated  cost  and  cost  realism-- 
namely  that  a  Government  estimated  cost  advantage  of  12  percent 
would  be  offset  by  a  cost  realism  difference  of  13  percent. 

By  calculating  the  tradeoffs  for  a  number  of  values  of 
technical  advantage,  tradeoff  curves  can  be  generated.  This  is 
illustrated  in  Figure  9.3,  which  plots  technical  advantage  as  a 
function  of  offsetting  cost  disadvantage  and  also  plots  tech¬ 
nical  advantage  as  a  function  of  offsetting  cost  realism  delta. 
The  same  spreads  and  weights  as  in  the  above  example  are 
assumed.  The  tradeoff  curve  for  Government  estimated  cost  and 
cost  realism  is  illustrated  in  Figure  9.4. 

It  should  be  noted  that  since  the  scoring  rules  and  weights 
must  be  established  before  the  solicitation  is  issued,  appro¬ 
priate  analyses  and  tradeoffs  must  also  be  done  before  the 
solicitation  is  issued. 

An  approach  to  conducting  the  cost  and  technical  tradeoffs 
is  discussed  next.  This  approach  is  based  on  developing 
"indifference  envelopes"  and  is  discussed  first  for  Government 
estimated  cost  and  then  for  cost  realism. 

Government  Estimated  Cost  Indifference  Envelope 

An  important  issue  facing  the  cost  evaluation  team  is  how 
much  should  the  Government  be  willing  to  pay  for  technical 
superiority?  For  example,  if  offeror  A  is  10  percent  tech¬ 
nically  better  than  B,  should  the  Government  pay  1  v.  percent 
more?  Should  they  pay  20  percent  more?  Answers  to  questions  of 
this  type  can  be  used  to  develop  "indifference  envelopes "--the 


9-5 


GOVERNMENT  ESTIMATE  COST  REALISM 


o  . — 

CNJ 


CD 

O 


C£L 

<c 


on 

o 

on 


o  u. 

T— ♦  U- 

o 


o 

c 


o 


o 


zc 


o  h- 


cn 


<1>  tO  4-> 
ZS  O  i/> 
L»j 


T3 

C 

ra 


QJ  OJ  -p 
-c  4->  c: 
■P  (t>  OJ 

o  E  E 
c 
s- 


4->  4-J 
lA  I/I  QJ 
O  LlJ  > 

o 


c 

.c 

u 

0/ 


o 

4->  O 
C  I 
a»  i 
E  i/> 
C  “D 
U  ro 
<D  QJ 
>  S- 
O  CL 
O  t/0 


o  a; 

cn 
OJ  ro 
■o  -*-> 
ro  c 
S-  ro 
h-  > 

■a 
c  <o 
ro  CO 

<_>  *r— 

O 

a; 

i-  -M 
O  ro 
S-  E 

<U  *r- 


C0 

o 

C-> 


ro 


a; 

V- 

3 

CD 


to 


W 

3 

to 

to 

<: 


|l 

X 


o 


o 

ro 


o 

CSJ 


(o  =  a  ‘a  .»cu3j.jo)  (%)  a  oi  3aiivi3h 

V  y0tf3J30  JO  30V1NVA0VSIQ  31VWI1S3  J.N3WNH3A09 


9-6 


combinations  of  technical  c a p a b i 1 i t y / c om p e t e n c e  and  cost  for 
which  the  cost  evaluation  team  and  the  technical  evaluation  team 
(which  will  be  referred  to  as  evaluation  teams)  would  be 
indifferent  or  uncertain  as  to  which  contractor  should  receive 
the  award.  This  concept  is  illustrated  below. 


To  construct  an  indifference  envelope  for  any  particular 
source  selection,  the  cost  evaluation  team  should  try  to  answer 
the  following  questions.  Given  that  offeror  A  has  a  certain 
technical  advantage  over  B  (A  has  a  technical  score  X  percent 
higher  than  B),  then: 

Up  to  what  additional  cost  should  the  Government 
almost  certainly  be  willing  to  pay  A  relative  to  B  for 
A's  technical  advantage? 

Beyond  what  additional  cost  should  the  Government 
almost  certainiy  prefer  to  award  to  B,  despite  A's 
technical  advantage? 

Generally,  the  answers  to  these  two  questions  will  not  be  the 
same  because  there  will  be  a  range  of  costs  which  the  evaluation 
teams  will  be  indifferent  about  or  uncertain  as  to  whether  to 
award  to  offeror  A  or  B.  Therefore,  answers  to  these  questions 
can  be  used  to  establish  lower  and  upper  bounds,  respectively, 
for  an  "indifference  envelope".  For  example,  if  offeror  A  has  a 
10  percent  technical  advantage  over  B,  the  evaluation  te  is  may 
prefer  to  award  the  contract  to  A  even  if  4  has  up  to  a  7  to  8 
percent  cost  disadvantage.  However,  if  A  has  a  17  percent  or 
more  cost  disadvantage,  they  may  prefer  to  award  the  contract  to 
offeror  B.  In  between,  they  may  be  indifferent  or  uncertain  as 
to  which  offeror  to  prefer.  For  other  degrees  of  technical 
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advantage,  the  evaluation  teams  would  make  similar  determi¬ 
nations  . 

An  example  of  an  indifference  envelope  that  has  some  plau¬ 
sibility  is  shown  in  Figure  9.5.  Below  the  envelope,  offeror  A 
is  generally  favored  and,  above  the  envelope,  B  is  generally 
favored.  If  the  lower  bound  is  placed  too  low,  offeror  A  could 
lose  even  though  they  are  very  superior  technically  at  relative¬ 
ly  little  added  cost  (e.g.,  20  percent  better  technically  at 
only  10  percent  higher  cost,  but  offeror  A  loses).  If  the  upper 
bound  is  placed  too  high,  offeror  A  could  win  even  though  the 
price  for  being  very  superior  is  rather  high  (e.g.,  20  percent 
better  technically,  but  40  percent  higher  in  cost,  yet  offeror  A 
still  wins).  In  sum,  technical  superiority  should  be  rewarded 
but  not  at  too  high  a  price. 

The  cost  evaluation  teams  can  use  the  indifference 
envelope,  which  represents  the  evaluation  teams  best  estimate  of 
acceptable  technical  and  Government  estimated  cost  tradeoffs,  to 
determine  the  Government  estimated  cost  criterion  weighting  in 
the  following  way.  Government  estimated  cost  and  technical 
tradeoff  curves  like  those  shown  earlier  in  Figure  9.3  can  be 
developed  for  several  different  Government  estimated  cost 
weights.  The  weights  can  be  conveniently  expressed  in  terms  of 
the  ratio  of  the  weight  for  the  Government  estimated  cost  score 
to  the  weigh"  for  the  technical  score.  For  example,  a  Govern¬ 
ment  estimated  cost  weight  of  20  percent  and  a  technical  weight 
o r  60  percent  results  in  a  ratio  (W)  equal  to  0.33.  The  trade¬ 
off  curves  for  several  values  of  W  can  then  be  superimposed  on 
the  graph  of  the  indifference  envelope.  This  is  illustrated  in 
Figure  9.6.  An  acceptable  weighting  would  be  one  where  the 
tradeoff  curve  falls  well  within  the  indifference  envelope  as, 
for  example,  the  tradeoff  curve  based  on  W  =  0.33  shown  in 
Figure  9.6. 

The  picture  presented  by  Figure  9.6  is  simplified  since  the 
tradeoff  curves  were  computed  assuming  first  that  offeror  A  has 
the  highest  possible  technical  score  (e.g.,  60  points  out  of  a 
possible  60  poinLs)  and  second  that  offeror  B  has  the  highest 
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TECHNICAL  ADVANTAGE  OF  OFFEROR  A  RELATIVE  TO  B  (%) 


Figure  9.5  Government  Estimated  Cost  Indifference  Envelope 
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Note:  W  is  the  ratio  of  the  Gover  estimated  cost 
weight  to  the  technical  .  Offeror  A  is 

assumed  to  receive  a  maximum  technical  score, 
and  Offeror  B  is  assumed  to  receive  a  maximum 
Government  estimated  cost  score. 


Figure  9.6.  Government  Estimated  Cost  Indifference  Envelope/ 
Tradeoff  Graph 


possible  Government  estimated  cost  score,  i.e.,  B's  Government 
estimated  cost  is  the  lowest  Government  estimate  cost  (Delta  = 
0)  giving  a  maximum  score  of  10.  These  are  the  baseline  assump¬ 
tions.  Assuming  that  B's  technical  score  is  not  likely  to  be 
less  than  70  percent  of  the  maximum  possible  score  (for  example, 
70  percent  might  represent  a  point  below  which  an  offeror  should 
be  disqualified),  a  lower  bound  tradeoff  curve  can  be  computed 
as  shown  in  Figure  9.7.  Assuming  that  B's  Delta  is  no  more  than 
5  percent  (i.e.,  there  is  an  offeror  C  whose  Government  esti¬ 
mated  cost  is  as  much  as  5  percent  less  than  B's),  a  tradeoff 
curve  which  falls  between  the  baseline  and  lower  bound  can  be 
computed  as  shown  in  Figure  9.7.  Thus,  the  tradeoff  for  a 
particular  W  becomes  a  band  rather  than  a  line. 

Cost  Realism  Indifference  Envelope 

Another  important  issue  facing  the  cost  evaluation  team  is, 
given  that  an  offeror  is  technically  superior,  should  the 
Government  be  concerned  if  the  offeror  is  unrealistic?  For 
example,  if  offeror  A  is  10  percent  technically  better  than  B, 
should  the  Government  be  concerned  that  offeror  A  is  unrealistic 
by  a  factor  of  10  percent  when  compared  to  the  Government 
estimate?  Should  they  be  concerned  if  A  is  unrealistic  by  20 
percent?  Answers  to  questions  of  this  type  can  be  used  by  the 
evaluation  teams  to  develop  "indifference  envelopes"  in  the  same 
manner  as  discussed  for  Government  estimated  cost. 

To  construct  an  indifference  envelope,  the  evaluation  teams 
should  try  to  answer  the  following  questions.  Given  that 
offeror  A  has  a  certain  technical  advantage  over  B  (A  has  a 
technical  score  X  percent  higher  than  B),  then: 

Up  to  what  degree  of  unrealism  in  A's  offer  would 
the  Government  almost  certainly  accept  and  still  award 
the  contract  to  A? 

Beyond  what  degree  of  unrealism  in  A's  offer  would 
the  Government  almost  certainly  prefer  to  award  to  B? 

Answers  to  these  questions  can  be  used  to  establish  lower  and 
upper  bounds,  respectively,  for  an  indifference  envelope.  For 
example,  if  offeror  A  has  a  10  percent  technical  advantage  over 
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Note:  The  shaded  area  represents  a  Government  estimated  cost  weight 
to  technical  weight  ratio  of  W  =  0.33.  The  top  line  assumes 
a  maximum  technical  score  for  A  and  D  =  0  for  B,  as  in  the 
previous  figure.  The  middle  line  assumes  D  =  0  for  B,  and 
the  bottom  line  assumes  70%  of  maximum  technical  score  for  B 
and  D  =  0  for  B. 


Figure  9,7  Government  Estimated  Cost  Tradeoff  Band 
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B,  the  evaluation  teams  may  prefer  to  award  the  contract  to  A 
even  if  A  is  7  to  8  percent  unrealistic.  However,  if  A  is  more 
than  17  percent  unrealistic,  they  may  prefer  to  award  to  offeror 
B. 

An  example  of  an  indifference  envelope  that  has  some  plau¬ 
sibility  is  shown  in  Figure  9.8.  Below  the  envelope,  offeror  A 
is  generally  favored  and  above  the  envelope,  offeror  B  is 
generally  favored.  If  the  lower  bound  is  placed  too  low, 
offeror  A  could  lose  even  though  technically  they  are  very 
superior  with  only  a  slightly  unrealistic  offer.  If  the  upper 
bound  is  placed  too  high,  offeror  A  could  win  even  though  their 
offer  is  judged  to  be  very  unrealsitic. 

The  cost  evaluation  team  can  use  the  indifference  envelope 
to  determine  the  cost  realism  criterion  weighting  in  the  same 
manner  as  discussed  for  Government  estimated  cost.  Tradeoff 
curves  can  be  superimposed  on  the  graph  of  the  indifference 
envelope  as  illustrated  in  Figure  9.9.  An  acceptable  weighting 
would  be  one  where  the  tradeoff  curve  falls  well  within  the 
indifference  envelope  as,  for  example,  the  tradeoff  curve  based 
on  W  =  0.17  as  shown  in  Figure  9.9.  In  this  case,  W  is  the 
ratio  of  the  weight  for  the  cost  realism  score  (10)  to  the 
weight  for  the  technical  score  (60). 

The  picture  presented  by  Figure  9.9  is  simplified  since  the 
tradeoff  curves  were  computed  assuming  that  offeror  A  has  the 
highest  possible  technical  score  and  that  offeror  B  has  the 
highest  possible  cost  realism  score.  Assuming  that  B's  tech¬ 
nical  score  is  at  least  70  percent  of  the  maximum  possible  score 
and  that  B's  cost  realism  delta  (Delta')  is  no  more  than  5 
percent,  a  tradeoff  curve  band  can  be  computed  as  shown  in 
Figure  9.10. 

Government  Estimate/Cost  Realism  Indifference  Envelope 

The  above  discussion  of  tradeoffs  has  focused  on  cost  and 
technical  tradeoffs.  The  tradeoff  between  Government  estimated 
cost  and  cost  realism  was  not  explicitly  considered.  However  in 
establishing  the  indifference  curves  for  Government  estimated 
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Figure  9.8  Cost  Realism  Indifference  Envelope 
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Notes:  The  shaded  area  represents  a  cost  realism 

weight  to  technical  weight  ratio  of  W=0.17. 

The  top  line  assumes  a  maximum  technical  score 
forAand  D '  =0  for  B,  as  in  the  previous  figure. 
The  other  lines  assume  D=5%  for  B  and  assume 
70%  of  the  maximum  technical  score  for  B  and 
D 1 =0  for  B. 


Figure  9.10  Cost  Realism  Tradeoff  Band 


cost  and  cost  realism  as  a  function  of  technical  advantage, 
there  was  an  implicit  tradeoff  between  Government  estimated  cost 
and  cost  realism.  This  tradeoff  between  Government  estimated 
cost  and  cost  realism  should  be  made  explicit  and  checked  for 
reasonableness . 

Indifference  envelopes  can  be  developed  in  the  same  manner 
as  discussed  above  for  Government  estimated  cost  and  cost 
realism.  The  evaluation  teams  should  try  to  answer  the  follow¬ 
ing  questions.  Given  that  offeror  A  has  a  certain  Government 
estimated  cost  advantage  over  B  (e.g.,  A's  Government  estimated 
cost  score  is  10  percent  higher  than  B's),  then: 

Up  to  what  degree  of  unrealism  in  A’s  offer  would 
the  Government  almost  certainly  accept  and  still  award 
the  contract  to  A? 

-  Beyond  what  degree  of  unrealism  in  A's  offer  would 
the  Government  almost  certainly  prefer  to  award  to  B? 

Answers  to  these  questions  can  be  used  to  establish  lower  and 
upper  bounds,  respectively,  for  an  indifference  envelope.  For 
example,  if  offeror  A  has  a  10  percent  estimated  cost  advantage 
over  B,  the  evaluation  teams  may  clearly  prefer  to  award  the 
contract  to  A  even  if  A  is  9  percent  unrealistic.  However,  if  A 
is  more  than  13  percent  unrealistic,  chey  may  clearly  proper  to 
award  to  offeror  B. 

An  indifference  envelope  may  also  be  determined  from  the 
Government  estimated  cost  and  cost  realism  indifference 
envelopes.  For  example,  if  for  a  technical  advantage  of  15 
percent,  the  Government  estimated  cost  indifference  envelope 
lower  bound  is  11  percent  and  the  cost  realism  lower  bound  is  15 
percent,  then  these  values  are  one  point  on  the  Government 
estimate/cost  realism  indifference  envelope  lower  bound. 

An  indifference  envelope  is  shown  in  Figure  9.11  based  on 
the  Government  estimate  and  cost  realism  envelopes  shown  in 
Figure  9.5  and  9.8.  Below  the  envelope,  offeror  A  is  generally 
favored  and  above  the  envelope,  offeror  B  is  generally  favored. 
If  the  lower  bound  is  placed  too  low,  offeror  A  could  lose  even 
though  they  have  a  significant  cost  advantage  and  only  a 
slightly  unrealistic  offer.  If  the  upper  bound  is  placed  too 


9-18 


20 


0  5  10  15  20 

GOVERNMENT  ESTIMATE  ADVANTAGE  OF  OFFEROR  A  RELATIVE  TO  B  {%) 


Figure  9.11.  Government  Estimate/Cost  Realism  Indifference  Envelope 
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high,  offeror  A  could  win  even  though  their  offer  is  judged  to 
be  very  unrealsitic. 

The  cost  evaluation  team  can  use  the  indifference  envelope 
to  determine  the  relative  weights  of  Government  estimated  cost 
and  cost  realism  in  the  same  manner  as  discussed  for  Government 
estimated  cost  and  cost  realism.  Tradeoff  curves  can  be  super¬ 
imposed  on  the  graph  of  the  indifference  envelope  as  illustrated 
in  Figure  9.12.  An  acceptable  weighting  would  be  one  where  the 
tradeoff  curve  falls  well  within  the  indifference  envelope  as, 
for  example,  the  tradeoff  curve  based  on  R  =  0.67  as  shown  in 
Figure  9.12.  In  this  case,  R  is  the  ratio  of  the  weight  for  the 
Government  estimated  cost  score  to  the  sum  of  the  weights  for 
the  Government  estimated  cost  score  and  the  cost  realism  score. 
For  example,  a  Government  estimate  cost  weight  of  20  percent  out 
of  a  combined  Government  estimate  cost  and  cost  realism  weight 
of  30  percent  gives  a  ratio  of  0.67.  The  tradeoff  curves  in 
Figure  9.12  assume  a  Government  estimated  cost  spread  twice  the 
cc-st  realism  spread,  e.g.,  40  and  20  percent  spreads,  respec¬ 
tively.  (The  picture  presented  by  Figure  9.12  is  simplified 
since  the  tradeoff  curves  were  computed  assuming  that  offeror  A 
has  the  highest  possible  technical  score  and  that  offeror  B  has 
the  highest  possible  cost  realism  score.) 

The  tradeoff  curves  in  Figure  9.12  may  be  interpreted  as 
follows.  For  R  =  0.50  (equal  weighting  of  Government  estimated 
cost  and  cost  realism),  offeror  B  would  win  over  offeror  A  if 
B's  cost  was  10  percent  higher  than  A's,  but  A  was  unrealistic 
by  6  percent  or  more.  For  R  =  0.67,  B  would  win  if  B's  cost  was 
10  percent  higher  but  A  was  unrealistic  by  11  percent  or  more. 
For  R  =  0.75,  B  would  win  if  B's  cost  was  10  percent  higher  but 
A  was  unrealistic  by  16  percent  or  more.  he  trades  suggested 
by  the  curve  with  R  =  0.50  does  not  seem  reasonable.  For 
example,  it  suggests  that  the  Government  would  pay  20  percent 
more  rather  than  accept  an  offer  which  was  unrealistic  by  more 
than  9  percent.  The  other  two  tradeoff  curves  which  weight 
Government  estimated  cost  significantly  higher  than  cost  realism 
do  seem  reasonable.  The  cost  evaluation  team  and  SSA  must 
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REALISM  DELTA  (D*)  FOR  OFFEROR 
(Offeror  B,  D'  =  0) 


A# 


R  =  0.6 


GOVERNMENT  ESTIMATE  ADVANTAGE  OF  OFFEROR  A  RELATIVE  TO  B  {%) 
(Offeror  A,  D  =  0) 

Note:  R  is  the  ratio  of  the  Government  estimated 

cost  weight  to  the  sum  of  the  Government  estimated 
cost  and  cost  realism  weights 


Figure  9.12.  Government  Estimate/Cost  Realism  Indifference  Envelope/ 
Tradeoff  Graph 


develop  tradeoff  curves  like  these  and  decide  which  weight  ratio 
is  reasonable. 

Combined  Weights 

Appropriate  weights  for  Government  estimated  cost  and  cost 
realism  can  be  determined  as  described  above.  Once  these 
weights  are  determined  they  should  be  summed  to  determine 
whether  they  add  to  an  appropriate  total  weight  for  the  cost 
evaluation  (assuming  that  DTUPC  and  LCC  evaluation  criteria  are 
not  included  in  the  cost  evaluation).  For  example,  based  on 
Figures  9.6  and  9.9,  W’s  for  Government  estimated  cost  and  cost 
realism  of  0.33  and  0.17,  respectively,  appeared  to  be  appro¬ 
priate.  These  values  sum  to  0.50.  If  technical  is  weighted  60 
percent,  this  suggests  that  the  cost  evaluation  weight  would  be 
30  percent  (i.e.,  0.50  X  60).  If  the  overall  cost  evaluation 
weight  is  not  appropriate  then  the  Government  estimated  cost  and 
cost  realism  weights  can  be  adjusted  to  some  extent,  especially 
since  their  selection  is  somewhat  arbitrary.  Also,  if  necessary 
the  Government  estimated  cost  and  cost  realism  spreads  can  be 
adjusted  to  some  extent,  since  their  selection  is  also  somewhat 
arbitrary. 

A  computer  model  has  been  developed  for  generating  tradeoff 
curves  and  is  described  in  Appendix  F,  Cost  Realism  Computer 
Models.  Illustrative  tradeoff  curves  are  provided  for  a  broad 
range  of  values  for  the  spreads  and  weights. 
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SECTION  X 

ILLUSTRATIVE  EXAMPLE 


This  section  provides  an  illustrative  example  of  how  to 
apply  the  cost  evaluation  methodology  presented  in  the  pre- 
ceeding  sections.  A  full  scale  development,  cost  r eimburseab le 
competition  is  assumed.  Technical,  schedule  and  cost  risk  are 
assumed  to  be  medium.  The  hypothetical  activities  that  would  be 
followed  by  a  cost  evaluation  team,  along  with  the  SSA  and 
others  in  the  source  selection  organization  (collectively 
referred  to  as  the  SSA),  are  presented  below.  These  activities 
begin  with  preparation  of  the  source  selection  plan  and  continue 
through  proposal  evaluation  and  award. 

A.  SOURCE  SELECTION  PLAN 

First,  as  part  of  the  source  selection  plan  preparation, 
the  cost  evaluation  team  and  SSA  should  decide  what  factors  to 
include  in  the  cost  evaluation.  As  recommended  in  Section  V, 
both  cost  realism  as  well  as  Government  estimated  cost  should  be 
included  as  explicit  cost  evaluation  sub-criteria.  For  this 
example,  it  is  assumed  that  it  is  decided  not  to  include  DTUPC 
or  LCC  as  explicit  evaluation  sub-criteria  because  they  are 
judged  to  be  of  lesser  importance,  the  expense  (both  contractor 
and  Government)  to  prepare  and  evaluate  them  accurately  appears 
to. outweigh  the  benefits,  and  they  would  tend  to  "water  down” 
the  importance  of  the  Government  estimated  cost  and  cost  realism 
criteria . * 

Next,  the  cost  evaluation  team  and  SSA  should  establish  the 
scoring  rules  to  be  used  for  Government  estimated  cost  and  cost 
realism.  The  three  major  decisions  are:  (1)  the  type  of  scoring 
rule  (e.g.,  trapezoid  or  normal  curve)  (2)  the  spread  of  the 
scoring  rule;  and  (3)  the  sub-criteria  weights. 


*  The  cost  evaluation  team  may  still  want  to  suggest  in 
the  solicitation  that  the  offerors  discuss  their  DTUPC 
and/or  LCC  programs  and  how  they  intend  to  meet  their  cost 
goals . 
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The  triangle  and  the  trapezoid  scoring  rules  are  used  for 
Government  estimated  cost  and  cost  realism,  respectively,  for 
the  reasons  discussed  in  Section  VIII. 

The  spreads  of  the  Government  estimated  cost  and  cost 
realism  scoring  rules  should  be  determined  primarily  on  the 
basis  of  assurring  that  there  is  sufficient  differentiation 
among  offerors.  Consideration  should  also  be  given  to  the 
potential  uncertainties  in  the  Government  estimates,  and  the 
cost  realism  scoring  rule  spread  should  be  at  least  as  great  as 
these  uncertainties*. 

For  Government  estimated  cost,  a  spread  of  30  to  40  percent 
was  suggested  in  Section  VIII  as  reasonable  considering  some 
data  on  past  cost  evaluations.  The  differentiation  of  scores 
for  different  spreads  can  be  considered  by  examining  the  table 
on  page  8-11.  For  spreads  that  are  either  very  high  or  very 
low,  there  is  not  a  reasonable  dif f erentation  of  scores.  Based 
on  this  consideration,  as  well  as  the  likelihood  that  a  Govern¬ 
ment  estimated  cost  more  than  40  percent  above  the  lowest 
Government  estimate  would  be  unacceptable,  a  spread  of  40 
percent  is  selected. 

For  cost  realism,  a  spread  of  15  to  25  percent  was 
suggested  in  Section  VIII  as  reasonable  considering  some  data  on 
past  cost  evaluations  and  the  likely  uncertainty  in  the  Govern¬ 
ment  estimates.**  Based  upon  a  consideration  of  the  differen¬ 
tiation  of  scores  for  different  spreads,  the  fact  that  the 
program  is  medium  risk,  and  the  fact  that  offers  greater  than  20 
percent  from  a  carefully  prepared  Government  estimate  would 
certainly  raise  a  serious  question  of  cost  realism,  a  spread  of 
20  percent  is  selected. 

The  weights  for  the  Government  estimated  cost  and  cost 
realism  criteria  may  be  best  determined  by  first  developing 


*  As  discussed  in  Section  VIII,  unless  the  spread  in  the 
uncertainty  distribution  about  the  Government  estimate  is 
greater  than  the  spread  in  the  scoring  rule,  there  is  no  need  to 
determine  the  uncertainty  for  cost  realism  cost  scoring. 

**  For  this  illustration,  the  uncertainty  spread  is  assumed 
to  be  small  and  less  than  the  scoring  rule  spread. 
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indifference  envelopes  as  discussed  in  Section  IX.  The  cost 
evaluation  team  should  develop  indifference  envelopes  by  consid¬ 
ering  the  various  tradeoffs  between  Government  estimated  cost, 
cost  realism,  and  technical  capability  for  two  hypothetical 
offerors  (A  and  B).  It  is  assumed  that  the  cost  evaluation  team 
with  the  assistance  of  the  technical  evaluation  team  and  the  SSA 
develops  indifference  envelopes  like  those  shown  in  Figures  9.5, 
9.8  and  9.11.  The  cost  evaluation  team  should  next  develop  cost 
and  technical  tradeoff  curves  and  superimpose  them  on  the  indif¬ 
ference  envelope  graphs.  It  is  assumed  that  the  cost  evaluation 
team  develops  graphs  like  those  shown  in  Figures  9.6,  9.9  and 
9.12  and  that  they  and  the  SSA  conclude  that  the  weight  for 
Government  estimated  cost  should  be  0.33  times  the  technical 
weight  and  the  weight  for  cost  realism  should  be  0.17  times  the 
technical  weight.  Therefore,  Government  estimated  cost  is 
weighted  twice  as  heavily  as  cost  realism.  It  is  further 
assumed  that  they  and  the  SSA  have  decided  that  the  technical 
evaluation  weight  should  be  60  percent  of  the  total  evaluation 
score  and  that  the  total  cost  evaluation  weight  should  be  30 
percent.  Thus,  Government  estimated  cost  will  receive  20  per¬ 
cent  of  the  total  evaluation  score  and  cost  realism  10  percent. 
These  sum  to  30  percent  which  is  the  same  as  the  total  cost 
evaluation  weight  already  decided  upon.  If  the  sum  had  been 
different,  the  Government  estimated  cost  and  cost  realism 
weights  and  the  scoring  rule  spreads  could  be  adjusted. 

Also  during  this  early  phase  of  the  source  selection,  it  is 
assumed  that  the  cost  evaluation  team  develops  an  independent 
cost  estimate  of  the  system  to  indicate  areas  the  WBS  should 
highlight,  where  the  detailed  fact-finding  sh'-*,1d  be  directed, 
and  to  use  in  cross-checking  the  Government  est"'"  tes  that  will 
be  developed  after  fact-finding  for  each  offeror. 

B.  SOLICITATION 

The  cost  evaluation  team  needs  to  be  sure  that  the  solici¬ 
tation  clearly  informs  potential  offerors  of  the  cost  evaluation 
criteria  as  well  as  the  data  needed  to  support  the  cost  evalua- 
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tion.  They  can  do  this  by  including  in  the  RFP  or  RFQ  some  or 
all  of  the  clauses,  modified  appropriately,  that  were  suggested 
in  Section  VI,  Also,  the  cost  evaluation  team  should  specify  in 
the  RFP  a  WBS  tailored  to  the  particulars  of  the  procurement  and 
the  types  of  data  required.  The  WBS  should  introduce  consis¬ 
tency  among  the  offerors  and  highlight  those  areas  where  cost 
visibility  is  desired. 

C.  PROPOSAL  EVALUATION/GOVERNMENT  ESTIMATE 

After  receipt  of  the  proposals,  the  cost  evaluation  team 
should  review  the  cost  proposals  for  completeness  and  consis¬ 
tency.  Then  they  should  conduct  an  extensive  fact-finding  at 
each  of  the  offerors'  plants  which  provides  the  opportunity  to 
obtain  clarifications,  to  examine  additional  backup  data,  and  to 
request  that  the  offerors  justify  their  cost  estimates  based  on 
their  actual  experience  on  previous  programs. 

After  fact-finding  and  some  further  analysis  and  review, 
the  cost  evaluation  team  should  develop  a  realistic  Government 
estimate  for  each  offeror.  These  Government  estimates  are  based 
on  the  offeror’s  cost  proposal,  the  data  collected  during  fact¬ 
finding,  and  on  data  reviewed  by  DCAA  (e.g.,  labor  rates,  over¬ 
head  rates,  G&A  rates  and  vendor  quotes).  The  Government 
estimates  are  prepared  at  the  WBS  item  level — accepting  the 
offeror's  estimate  as  is  for  that  line  or  adjusting  the 
offeror's  estimate  based  on  an  analysis  of  the  offeror's 
history,  on  the  correction  of  any  errors  found,  and  on  the  use 
of  the  Government's  data  base  for  other  programs  when  the 
offeror's  history  is  lacking  or  not  relevant. 

D.  SCORING 

With  the  scoring  rules  and  weights  established  during  the 
preparation  of  the  source  selection  plan  and  the  Government 
estimates  for  each  offeror  developed  as  part  of  the  cost  propos¬ 
al  evaluations,  the  cost  evaluation  team  can  then  rate  or  score 
each  of  the  proposals  for  Government  estimated  cost  and  cost 
realism.  It  is  assumed  that  the  cost  evaluation  team  and  SSA 
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decide  to  use  numerical  scoring  because  it  provides  the  clearest 
understanding  of  the  relative  ranking  of  the  various  evaluation 
alternatives.  It  is  also  assumed  that  they  prepare  a  cost 
evaluation  summary  that  includes  a  narrative  discussion  and 
tabular  and  graphical  displays  of  the  scores  and  other  cost 
information . 

For  this  illustrative  example,  it  is  assumed  that  five 
offerors  submit  proposals  with  proposed  costs  as  shown  below. 
Government  estimated  costs  which  are  assumed  to  have  been 
developed  by  the  cost  evaluation  team  are  also  shown  below. 


OFFEROR 


PROPOSED  COST 
( $  Millions) 


GOVERNMENT  ESTIMATED 
COST  ($  Millions) 


It  is  convenient  to  calculate  the  Government  estimated  cost 
premiums  compared  to  the  lowest  Government  estimate  ((G-G*)/G*  % 
=  D)  and  the  degree  of  realism  of  each  offeror  ((C-G)/G  %  =  D*). 
The  cost  evaluation  team  should  make  these  calculations  and 
prepare  a  tabular  display  as  shown  below: 


ILLUSTRATIVE  EXAMPLE  FOR  FSD  PROGRAM 
(Dollars  in  Millions) 


OFFEROR 

OFFER 

GOV’T 

GOV’T  EST 

REALISM 

(C) 

ESTIMATE 

DELTA  IN  % 

DELTA  IN  % 

(G) 

(D) 

(D’) 

A 

$47 

48 

-2.: 

B 

47 

54 

12.5 

-13. ( 

C 

53 

50 

4.2 

+  6 .  ( 

D 

60 

65 

35.4 

-7.' 

E 

89 

70 

45.8 

+  27.: 
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The  cost  evaluation  team  may  also  want  to  prepare  two 
grahical  presentations  of  their  findings  as  shown  in  Figure 
10.1.  The  tabular  and  graphical  displays  provide  an  excellent 
summary  of  the  key  cost  evaluation  data. 

Next,  the  cost  evaluation  team  should  score  each  offer 
based  on  the  scoring  rules  established  earlier.  The  scoring 
rules  are  graphed  in  Figure  10.2.  The  scores  for  each  offeror 
can  either  be  read  off  the  graph  or  computed  using  the  computer 
model  described  in  Appendix  F.  The  scores,  on  a  scale  of  zero 
to  ten,  are  tabulated  below. 


SCORES  (UNWEIGHTED) 
GOVERNMENT 


OFFEROR 

ESTIMATED  COST 

COST  REALISM 

A 

10.0 

10.0 

B 

7.2 

4.5 

C 

9.1 

8.0 

D 

2.0 

7.2 

E 

1.0 

1.0 

These  scores  can  then  be  weighted  based  on  the  Government 
estimated  cost  and  cost  realism  weights  of  20  percent  and  10 
percent,  respectively,  of  the  total  evaluation  score.  This  is 
shown  below: 

WEIGHTED  SCORES 

TECH  ADV 


OFFEROR 

GOV'T  EST 
COST  (20  MAX) 

COST  REALISM 
(10  MAX) 

TOTAL  COST 
EVAL  (30  MAX) 

TO  EQUALIZE 
SCORE 

A 

20.0 

10.0 

30.0 

. ,  _ . 

B 

14.4 

4.5 

18.9 

23  -  26% 

C 

18.1 

8.0 

26.1 

7-9% 

D 

4.1 

7.2 

11.2 

45% 

E 

2.0 

1.0 

3.0 

82% 

Offeror  A  has  the  highest  cost  evaluation  score  but  offeror  C's 
score  is  close.  Also  shown  in  this  table  is  the  degree  of 
technical  advantage  that  offeror  B,  C,  D  and  E  would  have  to 
have  over  offeror  A  in  order  to  equalize  their  scores.  The 
technical  advantage  is  measured  in  terms  of  the  percentage  that 
B,  C,  D  and  E's  technical  score  must  exceed  A's  technical  score 
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GOVERNMENT  EST1MEATE  (G) 
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Figure  10.1.  Graphical  Presentation  of  Cost  Evaluation  Data 
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Figure  10.2.  Government  Estimated  Cost  and  Cost  Realism  Scoring 
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to  break-even  (the  range  results  from  the  assumption  that  the 
technical  sc  es  may  be  between  70  percent  and  100  percent  of 
the  maximum  technical  score). 

The  above  cost  evaluation  information  together  with  a 
narrative  summary  of  the  cost  evaluation  team  findings  should  be 
prepared  and  presented  to  the  SSA  and  other  appropriate  source 
selection  personnel. 

Should  the  Government  request  best-and-f ina 1  offers  (BAFOs) 
from  the  offerors,  then  the  cost  evaluation  and  scoring  must  be 
revised  based  on  the  BAFO  submittals. 
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SECTION  XI 

OTHER  METHODS  FOR  ACHIEVING  COST  REALISM 


The  previous  sections  have  discussed  how  the  source  selec¬ 
tion  cost  evaluation  process  can  encourage  greater  realism  in 
contractor  cost  proposals.  This  section  will  briefly  discuss 
additional  methods  for  encouraging  and  achieving  greater 
contractor  cost  realism.  These  methods  include  incentive  type 
contracts,  risk  analysis  by  contractors  in  the  preparation  of 
proposals,  past  cost  performance  as  a  specific  evaluation 
criterion,  and  development  of  realistic  Government  budgets.  The 
extent  to  which  one  or  more  of  these  methods  would  encourage 
greater  cost  realism  depends  upon  the  particular  procurement. 
Although  other  means  for  improving  cost  realism  in  contractor 
proposals  may  also  exist,  the  following  discussion  will  focus  on 
those  mentioned  above. 

A.  CONTRACT  TYPE 

The  Government  uses  different  types  of  contracts  in 
different  procurement  situations  in  an  effort  to  control  the 
costs,  schedules  and  other  aspects  of  a  procurement.  For  the 
most  part,  the  Government  procures  weapon  systems  through  two 
basic  contract  types:  cost  reimbursement  and  fixed  price. 

Cost  Reimbursement  Contracts 

The  Government  normally  awards  cost  reimbursement  contracts 
during  the  RDT&E  or  the  early  production  phases  of  a  weapon 
system  acquisition  program  when  either  the  system  or  the  manu¬ 
facturing  processes  required  to  produce  it  (or  both)  involve 
major  technological  innovations.  In  such  high  risk  situations, 
the  Government  may  determine  that  other  factors  (such  as  pursuit 
of  research  with  problematic  results  or  the  expansion  of  a 
system’s  performance  parameters)  have  greater  importance  than 


*  In  certain  cases,  four  other  contract  types  are  used: 
letter,  time  and  materials,  labor,  and  indefinite  delivery.  The 
last  three  have  little  applicability  to  weapon  systems  acquisi¬ 
tion  programs. 
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cost  alone  and  that  it  is  willing  to  share  the  risk.  Under 
these  high  risk  conditions  and  in  the  absence  of  a  cost  reimbur- 
seable  contract,  potential  contractors  would  probably  include  a 
significant  cost  reserve  to  cover  their  risk  under  a  fixed  price 
contract  or  decide  not  to  bid  the  solicitation.  Therefore,  cost 
reimburseable  contracts  are  an  important,  necessary  part  of  the 
acquisition  process. 

Various  versions  of  cost  reimburseable  contracts  have  been 
developed,  differentiated  primarily  by  the  degree  of  risk  the 
contractor  assumes.  The  least  risky  version  for  the  contractor 
is  the  cost-plus-fixed-fee  (CPFF)  contract,  in  which  the  Govern¬ 
ment  pays  all  allowable  costs  as  well  as  a  fixed  fee  for  per¬ 
formance  of  the  contract.  A  CPFF  contract  provides  no  specific 
motivation  to  a  contractor  to  prepare  a  realistic  cost  proposal. 
A  contractor  believing  that  the  lowest  bidder  will  generally  be 
favored,  all  else  being  equal,  may  submit  an  unrealistically  low 
offer  knowing  that,  if  he  wins,  the  Government  will  pay  all 
allowable  costs.  (Of  course,  there  is  the  limitation  that  the 
Government  program  office  may  not  be  able  to  fund  the  overruns 
and,  in  the  extreme  case,  the  program  might  even  be  cancelled 
due  to  overruns). 

In  order  to  improve  cost  control  and  share  some  of  the 
risk,  the  Government  often  uses  incentive  type  cost- 
reimburseab 1 e  c o n t r a c t s --c o s t - p 1 u s- i n c en t i v e- f e e  (CPIF)  and 
cost-plus-award-fee  (CPAF).  The  CPIF  contract  encourages  the 
contractor  to  control  costs,  since  the  Government  and  the 
contractor  share  allowable  costs  over  a  specific  target  cost; 
the  contractor  then  maximizes  his  fee  by  performing  below  target 
cost.  (However,  the  contractor  may  consider  trading  off  fee  for 
additional  revenue  above  the  target).  Therefore,  it  is 
generally  in  the  offeror’s  self-interest  to  propose  realistic 
costs.  An  unrealistically  low  offer  resulting  in  an  unrealisti¬ 
cally  low  (and  probably  unachievable)  target  cost  would  leave 
the  contractor  with  little  or  no  fee.  Alternatively,  an 
unrealistically  high  offer  would  place  a  contractor  at  a 
competitive  disadvantage  in  the  source  selection. 
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The  other  cost  reimbur seab le  incentive  arrangement  is  the 
CPAF  contract,  which  includes  both  a  negotiated  fixed  fee  and  an 
optional  award  amount  for  reaching  specified  management  or 
system  performance  objectives.  Like  CPIF,  CPAF  contracts  can  be 
structured  to  provide  offerors  with  an  incentive  to  propose 
real istic  costs. 

Fixed  Price  Contracts 

The  Government  prefers  to  use  fixed  price  contracts  after  a 
contractor  has  established  rate  production  of  an  item  or  when 
the  item  and  its  manufacture  involve  mature  (and  therefore 
predictable)  technologies.  In  such  situations,  the  Government 
encourages  the  contractor  to  reduce  costs  in  order  to  maximize 
profits.  The  offeror  will  tend  to  propose  realistic  costs 
because  failure  to  do  so  will  affect  his  profits  or  the  pro¬ 
bability  of  award  (too  low,  no  profits  or  a  loss;  too  high,  no 
contract).  Variations  of  the  fixed-price  contract  are  firm- 
fixed-price  (FFP)  and  fixed-price-incentive  (FPI).  Fixed-price 
contracts  encourage  the  contractor  to  propose  realistic  costs 
and  to  control  costs  during  contract  execution.  An  unrealisti¬ 
cally  low  offer  could  mean  not  only  little  or  no  profit,  but  a 
loss;  an  unrealistically  high  offer  would  place  a  contractor  at 
a  competitive  disadvantage  in  the  source  selection.  However, 
fixed  price  contracts  generally  are  not  suitable  for  RDT&E 
programs . 

In  conclusion,  incentive  type  cost-reimburseab 1 e  contracts, 
as  well  as  fixed  price  contracts,  may  encourage  contractors  to 
submit  realistic  cost  proposals.  However,  unless  the  Government 
structures  the  incentives  appropriately,  they  may  not  yield  the 
desired  results  and  the  offerors  may  "game"  them  to  the  Govern¬ 
ment's  disadvantage. 

B.  RISK  ANALYSIS 

A  risk  analysis  is  sometimes  required  by  the  Government  as 
a  part  of  a  proposal.  In  particular,  the  Air  Force  requires 
risk  analysis  for  major  program  source  selections  in  AFR-70-15. 


This  document  states  that  "offerors  must  be  required  to  submit  a 
risk  analysis"  and  that  they  "will  be  required  to  identify  the 
risk  associated  with  their  proposals  in  the  areas  of  cost, 
schedule,  or  performance,  together  with  realistic  approaches  for 
resolving  or  avoiding  the  identified  risks  (including  backup 
development  efforts)." 

A  risk  analysis  focuses  attention  on  the  risks  and 
uncertainties  of  a  particular  procurement.  It  increases  the 
probability  that  the  contractor  will  account  for  risk  during 
proposal  preparation  and  therefore  submit  a  more  realistic  cost 
proposal.  Furthermore,  the  contractor’s  risk  analysis  should 
help  the  Government  evaluate  the  cost  proposal’s  realism. 

Greater  use  of  risk  analysis  may  also  encourage  the  Govern¬ 
ment  to  budget  for  risk  through  the  use  of  appropriate  manage¬ 
ment  reserves  (e.g.,  the  Array's  Total  Risk  Assessing  Cost  Esti¬ 
mate  (TRACE)  concept).  While  management  reserves  will  not  in 
themselves  encourage  greater  contractor  cost  realism,  they  could 
help  preserve  a  program's  budget  when  a  contract  is  awarded  to 
an  offeror  whose  cost  proposal,  and  therefore  the  award,  is 
unrealistically  low.  Thus,  management  reserves  would  reduce 
some  of  the  undesirable  consequences  of  unrealistically  low  cost 
proposals . 

C.  PAST  COST  PERFORMANCE 

The  Government  generally  includes  past  contractor  perfor¬ 
mance  in  technical,  management,  cost,  or  other  areas  among  the 
source  selection  evaluation  criteria.  For  major  programs,  DOD 
Directive  4105.62  requires  that  past  performance  be  a  factor  in 
the  selection  decision.  Making  offerors’  recent,  relevant  cost 
experience  (actual  versus  proposed)  on  cost  re i mb ur se a b 1 e 
contracts  an  explicit  evaluation  criterion  should  encourage  them 
to  estimate  their  costs  more  realistically,  since  the  degree  of 
realism  in  their  present  offer  may  affect  their  ability  to  win 
future  contracts. 
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D.  REALISTIC  BUDGET  POLICY 

Realistic  budgeting  not  only  by  the  Services  and  OSD  but 
also  by  the  Congress  should  have  a  positive,  though  indirect, 
effect  on  offerors'  bids.  Most  offerors  know  either  exactly  or 
approximately  the  amount  budgeted  for  a  particular  program  and, 
independent  of  competitive  pressures,  may  perceive  (rightly  or 
wrongly)  that  proposals  over  some  amount  will  not  sell.  If  an 
offeror's  actual  cost  estimate  exceeds  this  amount,  he  will  be 
inclined  to  submit  a  proposal  for  less  (i.e.,  an  unrealistic 
cost  proposal).  Obviously,  the  Government  can  reduce  or  elimi¬ 
nate  this  problem  by  developing  realistic  budget  estimates.  This 
requires  that  the  Government  improve  its  ability  to  estimate 
budgets  well  in  advance  of  solicitation  release,  based  only  on 
partially  defined  requirements. 
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APPENDIX  A 
DEFINITIONS 


Definitions  of  some  important  terms  used  in  this  report  are 
provided  below. 


Cost  Analysis 

The  general  sense  of  cost  analysis  as  used  in  DOD  is  the 
development  and  application  of  concepts  and  techniques  for 
assessing  the  costs  of  proposed  military  systems  and  capabili¬ 
ties.  Cost  analysis  often  involves  the  development  and  applica¬ 
tion  of  cost  estimating  relationships  (CERs)  based  on  the 
relationship  between  historical  costs  for  systems  or  components 
of  systems  and  various  technical,  performance  and  other  charac¬ 
teristics.  The  FAR  and  ASPM  No.l*  define  cost  analysis  in  a 
somewhat  different  sense  as  follows: 

The  review  and  evaluation  of  a  contractor's  cost  or 
pricing  data  and  of  the  judgmental  factors  applied  in 
projecting  from  the  data  to  the  estimated  costs  in 
order  to  form  an  opinion  leading  to  a  position  on  the 
degree  to  which  the  contractor's  proposed  costs 
represent  what  contract  performance  should  cost, 
assuming  reasonable  economy  and  efficiency.  It 
includes  appropriate  verification  of  cost  data, 
evaluation  of  specific  elements  of  costs  and 
projection  of  these  data  to  determine  the  effect  on 
price  factors  like  cost  necessity,  allowance  for 
contingencies,  and  the  basis  used  for  allocation  of 
overhead  costs. 

Cost/Price 

In  this  report  we  use  the  terms  cost  and  price  inter¬ 
changeably.  The  distinction  that  is  often  made  between  the 
terms  is  that  price  is  cost  plus  a  fee  or  profit.  However,  a 
contractor's  "price”  may  be  considered  the  Government's  "cost." 

The  ASPM  No.  1  defines  price  as  "A  monetary  amount  given, 
received  or  asked  in  exchange  for  property  or  services, 


*  Armed  Services  Procurement  Regulations  Manual  for  Con¬ 
tract  Pricing  (ASPM  No.  1),  Office  of  the  Secretary  of  Defense, 
September  1975. 
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expressed  in  terms  of  a  single  item  or  unit  of  measure  for  such 
property  or  services." 

Cost  Realism 

In  this  report  we  define  cost  realism  as  (1)  in  general, 
that  part  of  the  source  selection  process  directed  at  assuring 
the  reasonableness  and  def ensibi lity  of  contractors'  cost  pro¬ 
posals  and  (2)  more  specifically,  an  evaluation  criterion  stated 
in  the  solicitation  which  compares  the  contractor's  proposed 
cost  with  a  detailed  Government  estimate  for  each  contractor  and 
then  scores  the  degree  of  realism. 

The  terms  cost  reasonableness,  def ensibi 1 ity ,  credibility 
and  validity  often  have  the  same  meaning  as  cost  realism, 
depending  on  the  context. 

Government  Estimated  Cost 

The  "Government  estimated  cost"  and  "Government  estimate" 
are  terms  used  in  this  Handbook  to  refer  to  the  likely  cost  that 
will  be  incurred  as  the  result  of  awarding  a  particular  con¬ 
tract.  The  likely  cost  is  determined  by  a  detailed  Government 
estimate  for  each  offeror  (i.e.,  cost  analysis  in  the  sense  of 
the  FAR  and  ASPM  No.  1  ).  Parametric,  analogy  and  other  tech¬ 
niques  of  cost  analysis  which  utilize  cost  data  from  a  number  of 
sources  (e.g.,  actual  costs  for  similar  systems  and  industry 
wide  factors)  may  be  used  to  supplement  the  estimates  based  on 
specific  contractor  cost  data. 


Indifference  Envelope 

An  indifference  envelope  is  the  combinations  of  technical 
capability  and  cost  for  which  the  cost  evaluation  team  and  the 
technical  evaluation  team  would  be  indifferent  or  uncertain  as 
to  which  contractor  should  receive  the  award. 

Normal  Curve 

The  normal  curve  is  a  symmetrical  bell-shaped  curve  of  a 
normal  distribution  whose  "spread"  is  determined  by  the  standard 
deviation . 
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Offerors/Contractors 

These  terms  are  often  used  interchangeably  in  this  report. 
A  private  company  awarded  a  contract  by  the  Government  is  a 
"contractor.”  Ar.  offeror  is  a  contractor  (or  a  potential 
contractor)  who  submits  a  proposal  or  offer  to  the  Government  in 
response  to  a  Government  solicitation. 


Price  Analysis 

The  FAR  and  ASPM  No.  1  make  a  distinction  between  price 
analysis  and  cost  analysis.  They  define  price  analysis  as 
follows : 

The  process  of  examining  and  evaluating  a  prospective 
price  without  evaluation  of  the  separate  cost  elements 
and  proposed  profit  of  the  individual  offeror  whose 
price  is  being  evaluated.  It  may  be  accomplished  by  a 
comparison  of  submitted  quotations,  a  comparison  of 
price  quotations  and  contract  prices  with  current 
quotations  for  the  same  or  similar  items,  the  use  of 
rough  yardsticks  (dollars  per  pound,  for  instance),  or 
a  comparison  of  proposed  prices  with  independently 
developed  Government  estimates. 

Scoring  Rule 

A  scoring  rule  is  a  relationship  which  specifies  the  score 
asssociated  with  a  particular  evaluation  criterion  such  as  the 
Government  estimated  cost.  The  relationship  may  be  expressed  in 
equation  or  graphical  form.  Triangle,  trapezoid  and  normal 
curve  shaped  scoring  rules  are  recommended  in  this  Handbook. 


Spread 

Scoring  rules  can  be  narrow  or  wide  depending  upon  the 
degree  of  "spread".  This  Handbook  characterizes  spread  in  terms 
of  S  for  triangle  or  trapezoid  and  twice  the  standard  deviation 
(2a)  for  a  normal  curve. 


Technical  Advantage 

The  technical  advantage  of  one  contractor  relative  to 
another  is  determined  by  the  technical  evaluation  scores 
developed  for  each  contractor. 
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APPENDIX  B 

ACRONYMS  AND  SYMBOLS 


ASPM 

BAFO 

C 

D 

D  1 

DAR 

DCAA 

DCAS 

DOD 

DTUPC 

FAR 

G 

G* 

IFB 

LCC 

NAVAIR 

NAVELEX 

NAVSEA 

PCO 

PM 

R 

RDT&E 

RFP 

RFQ 

S 

SSA 

SSAC 

SSEB 

SSP 

W 


Armed  Services  Procurement  Regulation 
Manual  for  Contract  Pricing 

Best-and-Final  Offer 

Contractor  Offer 

Percentage  difference  between  G  and  G* 

Percentage  difference  between  C  and  G 

Defense  Acquisition  Regulations 

Defense  Contract  Audit  Agency 

Defense  Contract  Administration  Service 

Department  of  Defense 

Design-to-Unit  Production  Cost 

Federal  Acquisition  Regulations 

Government  Estimated  Cost 

Lowest  Government  Estimated  Cost 

Invitation  for  Bids 

Life  Cyle  Cost 

Naval  Air  Systems  Command 

Naval  Electronic  Systems  Command 

Naval  Sea  Systems  Command 

Procurement  Contracting  Officer 

Project/Program  Manager 

Ratio  of  Government  estimated  cost  weight 
to  sum  of  Government  estimated  cost  and 
cost  realism  weights 

Research,  Development,  Test  &  Evaluation 
Request  for  Proposals 
Request  for  Quotations 

Spread  or  point  at  which  the  cost  realism 
score  reaches  a  minimum 

Source  Selection  Authority 

Source  Selection  Advisory  Council 

Source  Selection  Evaluation  Board 

Source  Selection  Plan 

Ratio  of  Government  estimated  cost  or  cost 
realism  weight  to  technical  weight 
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WBS  Work  Breakdown  Structure 

a  Standard  Deviation 
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APPENDIX  C 

POLICY  DOCUMENT  EXCERPTS 


A  number  of  policy  documents  were  reviewed  and  excerpts 
related  to  source  selection-cost  evaluation  are  provided  in  this 
Appendix.  The  DAR  has  been  superseded  by  the  FAR;  the  wording 
in  the  FAR  may  not  be  exactly  the  same  as  the  DAR  excerpts  in 
this  Appendix. 

Excerpts  are  provided  in  the  following  order: 


Page 

Department  of  Defense 

DOD  Directive  4105.62  C-2 

Defense  Acquisition  Regulation  (DAR)  C-5 


Navy 

NAVMAT  Instruction  4200.49  C-12 

NAVELEX  Instruction  4200. 12C  C-14 

NAVAIR  Instruction  4200.24  C-17 

NAVAIR  Instruction  4200.27  C-19 

Army 

AMC  Regulation  715-1  C-21 

Army  Regulation  715-6  C-23 

AMC  Supplement  1  to  AR  715-6  C-24 

DARC0M  Pamphlet  715-3  C-25 

DARC0M  Circular  715-8-80  C-33 

Air  Force 

Air  Force  Regulation  70-15  C-34 

Air  Force  Regulation  80-15  C-43 
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DUD  Oi recti ve  4lUb.b2 

Selection  or  Contractual  Sources  tor  fajor  Defense  Systems 
b  January  197b 

Excerpts  related  to  cost  evaluation: 

-  I.  ...policy  for  tne  competitive  solicitation,  evaluation,  and  selection 
uf  contractual  sources  for  tne  acquisition  of  in  a j or  Defense  systems  in 
accordance  witn  tne  Dasic  philosophy  outlined  in  reference  (a).... 
(reference  (a)  is  DODD  500U.1) 

-  III.  Policy.  The  following  policy  snail  govern  the  solicitation, 
evaluation  ano  source  selection  process  as  conducted  by  DOD  Components. 

-  III. A.  oeneral 

1.  Tne  prime  ODjectives  or  tne  process  are  to  (a)  select  the  source 
wnose  proposal  nas  tne  mghest  degree  of  realism  and  credibility  and  whose 
performance  is  expected  to  best  meet  Government  objectives  at  an  affordable 
cost .... 

2. b.  In  addition,  the  solicitation  should  De  so  structured  so  as  to 
reflect  tne  mission  need,  schedule,  cost,  capability  objectives,  and 
operating  constraints  but  not  technical  approach  or  main  design  features. 

3.  Each  DUD  Component  shall  develop,  and  consistently  apply, 
procedures  wnicn  create  the  environment  for  an  impartial,  balanced  and 
realistic  appraisal  of  all  proposals  submitted. 

-  I1I.C.  Sol lcitation 

1.  Tne  Source  Selection  Plan.  Prior  to  tne  issuance  of  the 
solicitation,  a  Source  Selection  Plan  snail  be  approved  by  the  Source 
Selection  Authority.  Tne  plan  will  include  tne  evaluation  criteria  in 
relative  order  of  importance.  In  developing  tne  criteria  a  proper  balance 
snail  be  established  among  tecnnical,  cost,  schedule,  management  and  other 
factors . 

2.  Preparation  of  the  Solicitation. 

a.  The  solicitation  snould  be  designed  so  as  to  minimize  competitor 
and  Government  expense  incident  to  noth  preparation  and  response 
tnereto .... 

c.  It  is  essential  tnat  proposals  (including  revisions  thereto)  be 
submitted  on  a  oasis  wmch  permits  verification  of  their  cost  credibility. 
To  facilitate  this  objective  and  lay  the  groundwork  for  possible 
elimination  of  unrealistic  cost  proposals,  as  provided  herein,  every 
solicitation  shall  advise  each  competitor  that  his  proposal  is  presumed  to 
represent  nis  best  efforts  to  respond  to  tne  solicitation  and  any  apparent 
inconsistency  Detween  promised  performance  and  cost  or  price  should  be 
explained.  Furtner,  the  solicitation  snould  advise  tnat  any  significant 
inconsistency,  if  unexplained,  raises  a  fundamental  issue  of  the 
competitor's  understand i ng  of  the  work  required  and/or  of  his  ability  to 
perform  the  contract,  and  may  be  grounds  for  rejection  of  the  proposal  ,  or 
a  determination  of  nonresponsioi I ity.  Tne  solicitation  snail  advise  that 
tne  burden  of  proof  of  cost  credibility  rests  with  tne  offeror. 

f.  Trie  solicitation  snail  identify  tne  evaluation  criteria  against 
wnicn  all  proposals  will  be  evaluated.  These  criteria  snould  be  carefully 


structured  to  assure  emphasis  on  critical  items  or  areas. 

(1)  Such  criteria  shall  set  forth  the  relative  importance  among 
technical,  costs  (development  versus  production  versus  operational  and 
support  costs),  schedule,  management  and  other  factors  as  set  forth  in  the 
Source  Selection  Plan. 

-  III.D.  Evaluation 

1.  The  evaluation  shall  assess  both  the  proposal  and  the  capability  of 
each  competitor  to  successfully  accomplish  the  program  under  consideration. 
Past  performance  of  each  competitor,  when  it  is  relevant  to  the  anticipated 
procurement,  should  be  a  factor  in  the  selection  decision  to  the  extent 
indicated  in  the  solicitation. 

2.  The  primary  goal  of  evaluation  is  to  determine,  for  each 
competitor,  the  most  likely  outcome  in  terms  of  system  characteristics, 
costs  (development,  production,  and  support),  and  schedules. 

5.  This  paragraph  and  its  subparagraphs  establish  a  four  step 
process.... 

5.b.(3)(a) 

Prior  to  contractor  selection,  meaningful  discussions  will  be 
conducted  with  all  remaining  offerors  in  connection  with  their  respective 
cost/price  proposals,  either  on  an  element-by-element  basis  or  in  their 
entirety....  The  discussions  of  the  cost/price  proposals  may  include  (a) 
cost  realism  of  the  offeror's  proposal,  (b)  rectification  and/or  correction 
of  inconsistencies  or  mathematical  errors,  (c)  correlation  of  elements  of 
cost  with  their  respective  technical  efforts,  and  (d)  discussion  necessary 
to  ensure  a  complete  understanding  of  the  Government's  requirements  and 
what  is  being  offered,  including  delivery  schedules,  trade-offs  among 
performance ,  design  to  cost,  life  cycle  cost,  and  logistic  support  factors, 
and  other  contract  terms. 

5.b.(3)(d) 

Any  cost/price  changes  made  during  the  discussion  period 
including  the  finally  revised  proposal  must  be  fully  substantiated  by  the 
offeror.  Supporting  data  must  provide  traceability  to  the  causative 
technical  ,  business,  or  financial  conditions  that  brought  about  the  change. 
In  order  to  help  identify  "buy-ins",  lump  sum  reductions  in  cost/price 
should  not  be  accepted  without  supporting  rationale. 

5.b.(4)(b) 

Proposals  unrealistic  in  terms  of  technical  or  schedule 
commitments  or  unrealistically  low  in  cost  or  price  will  be  deemed 
reflective  of  an  inherent  lack  of  technical  competence  or  indicative  of 
failure  to  comprehend  the  complexity  and  risks  of  the  contract  requirements 
and  may  be  grounds  for  rejection  of  the  proposal. 

5.b.(4)(c) (4) 

Negotiations  after  selection  shall  not  involve  material  changes 
in  the  Government's  requirements  or  the  contractor's  proposal  which  affect 
the  basis  for  source  selection.  In  the  event  that  such  changes  are  desired 
by  the  Government,  the  competition  will  be  reopened  in  accordance  with 
existing  ASPR  requi rements. 

8.  DOD  Components  shall  develop  an  independent  cost  estimate  to  assist 
in  determining  the  most  probable  costs  (developmental,  production,  and 
support;  especially  the  ability  to  meet  the  design  to  cost  objectives)  of 
each  competitor's  proposal.  Parametric  cost  estimating  techniques  or 
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similar  approaches  should  be  used  to  tne  extent  practical  to  determine  the 
reasonabl eness  of  tnese  costs.  The  Source  Selection  Authority  shall  base 
his  selection  on  what,  in  his  best  judgment,  is  the  most  probable  outcome 
(product  performance,  cost  and  schedule)  for  each  proposal. 

9.  Unrealistically  low  cost  estimates,  initially  or  subsequently,  may 
be  grounds  for  eliminating  a  proposal  from  competition  either  on  the 
grounds  that  the  competitor  does  not  understand  the  requirartent  or  that  he 
has  made  an  improvident  proposal.  The  burden  of  proof  of  cost  credibility 
rests  with  the  competitors.  If  the  Source  Selection  Authority  determines  a 
competitor's  estimates  to  be  unexpl ainably  low,  that  competitor  may  be 
eliminated  from  the  competition....  Notice  to  this  effect  should  be  set 
forth  in  the  solicitation. 
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Defense  Acquisition  Regulation  (DAK) 

Excerpts  related  to  cost  evaluation: 

PAR  Section  III:  Procurement  by  Negotiation 
Part  8-  Price  Negotiation  Policies  and  Techniques 
-  3-801  Basic  Policy 

3-801.1  General.  It  is  tne  policy  of  the  Department  of  Defense  to 
procure  supplies  and  services  from  responsible  sources  at  fair  and 
reasonable  prices  calculated  to  result  in  the  lowest  ultimate  overall  cost 
to  the  Government.  Good  pricing  depends  primarily  upon  the  exercise  of 
sound  judgment  Dy  a  I  I  personnel  concerned  with  the  procurement. 

3-801.2  Responsibility  of  Contracting  Officers. 

(a)  ...Tne  contracting  officer  shall  avail  himself  of  all  appropriate 
organizational  tools  such  as  tne  advice  of  specialists  in  the  fields  of 
contracting,  finance,  law,  contract  audit,  packaging,  engineering,  traffic 
management,  and  price  analysis. 

(b)  ...Tnus,  determination  of  tne  suitability  of  the  contract  price  to 
tne  Government  always  remains  tne  responsibility  of  tne  contracting 
officer.  For  certain  acquisitions,  it  may  be  necessary  to  convene  a  formal 
"Snould  Cost"  (see  1-337)  team  of  specialists  to  evaluate  the  contractor's 
cost  projections,  supporting  standards,  and  otner  in-plant  management, 
operational,  and  performance  practices,  on  wnicn  cost  projections  are 
based . 

(d)  Pricing  based  on  cost  analysis  involves,  among  otner  things,  an 
appraisal  of  estimates  of  costs  expected  to  be  incurred  in  the  future.  The 
accounting  projection  of  trends  based  on  cost  or  pricing  data,  togetner 
witn  any  known  changes  therein,  is  only  one  method  of  conducting  tnis 
appraisal  ,  others  being: 

(l)  an  engineering  appraisal  of  tne  need  for  the  estimated  labor 
and  material  costs  and  of  tooling  and  facilities,  and  tne  reasonableness  of 
scrap  and  spoilage  factors,  and 

(n)  tne  preparation  of  independent  estimates  by  competent 
tecnnica I  personnel  . 

Occasionally,  differences  of  opinion  will  exist  not  only  on  the 
reasonableness  of  cost  projections  but  also  on  the  accounting  techniques  on 
which  they  are  based....  Tne  contracting  officer  is  responsible  for  tne 
exercise  of  tne  requisite  judgments  and  is  solely  responsible  for  the  final 
pricing  decisions. 

3-801.4  Responsibi I ity  of  Pricing  Personnel  in  Purchasing  Office. 

(a)  Tne  contract  pricing  team  available  to  support  the  PCO  in  tne 
review  and  analysis  of  pricing  proposals  includes  tne  price  analyst, 
negotiator,  buyer,  project  engineer,  and  liaison  auditor  at  tne  purchasing 
office. 

(b)  Tne  price  analyst  or  negotiator  supporting  tne  contracting  officer 
may  oe  designated  to  develop  a  Government  pricing  objective  prior  to  the 
negotiation.  Tnis  includes  tne  responsibi I ity  for  determining  the  extent 
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of  advice  required  from  otner  specialists,  requesting,  obtaining,  and 
considering  such  advice,  and  for  consolidating  pricing  data,  including  cost 
and  price  analyses,  historical  cost  or  pricing  data,  independent  government 
cost  estimates,  economic  analyses  and  the  like.  The  advice  and  assistance 
of  the  price  analyst/negotiator  should  always  be  obtained  when  complex 
pricing  techniques  are  indicated,  including  the  use  of  contract  types 
involving  skillfull  balancing  of  price,  cost  and  performance  incentive 
arrangements .  In  many  instances,  he  will  be  in  the  best  position  to 
conduct  the  price  negotiation. 

3-801.5  Responsibility  of  Field  Pricing  Support  Personnel. 

(a)  Field  Pricing  Support.  Field  pricing  support  involves  analysis  of 
the  contractor's  pricing  proposal  by  any  or  all  of  the  field  technical  and 
professional  specialists  including,  but  not  limited  to,  the  Plant  Rep/ACO; 
contract  auditor;  price  analyst;  quality  assurance  personnel,  engineers; 
and  legal  and  small  busines  specialists.  The  Plant  Rep/ACO  is  the  team 
manager  for  all  PCO  requests  for  field  pricing  support.  Therefore  the  PCO 
shall  send  all  requests  for  field  pricing  support  to  the  cognizant  field 
contract  administration  activity;  generally,  the  Plant  Representative 
(Plant  Rep)  for  the  Services  and  the  Administrative  Contracting  Officer 
(ACO)  for  OCAS(OSA).  However,  there  shall  be  no  constraints  on  the  line  of 
communication  with  the  PCO/ ACO  and  contract  auditor  interface. 

-  3-806  Cost,  Profit,  and  Price  Relationships. 

(a)  When  the  products  are  sold  in  the  open  market,  costs  are  not 
necessarily  the  controlling  factor  in  establishing  a  particular  seller's 
price.  Similarly  where  competition  may  be  ineffective  and  lacking, 
estimated  costs  plus  estimated  profit  are  not  the  only  pricing  criteria. 

In  some  cases,  the  price  appropri atel y  may  represent  only  a  part  of  the 
seller's  cost  and  include  no  estimate  for  profit  or  fee,  as  in  research 
projects  where  the  contractor  is  willing  to  share  part  of  the  costs.  In 
other  cases,  price  may  be  controlled  by  competition  as  set  forth  in 
3-805.1.  The  objective  of  the  contracting  officer  shall  be  to  negotiate 
fair  and  reasonable  prices  in  which  due  weight  is  given  to  all  relevant 
factors . 

(b)  ...While  the  public  interest  requires  that  excessive  profits  be 
avoided,  the  contracting  officer  should  not  become  so  preoccupied  with 
particular  elements  of  a  contractor's  estimate  of  cost  and  profit  that  the 
most  important  consideration,  the  total  price  itself,  is  distorted  or 
diminished  in  its  significance.  Government  procurement  is  concerned 
primarily  with  the  reasonabl eness  of  the  price  which  the  Government 
ultimately  pays,  and  only  secondarily  with  the  eventual  cost  and  profit  of 
the  contractor. 

-  3-807  Pricing  of  Negotiated  Contracts. 

3-807.1  General.  Policies  set  forth  in  this  part  may  be  applied  in  a 
variety  of  ways  in  evaluating  proposals  and  negotiating  contract  prices. 
These  provisions  apply  equally  to  initial  and  subsequent  price 
negotiations . 

(a)  Definitions. 

(1)  Cost  or  pricing  data  consists  of  all  facts  existing  up  to  the 
time  of  agreement  on  price  which  prudent  buyers  and  sellers  would 
reasonably  expect  to  have  a  significant  effect  on  price  negotiations.  Cost 
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or  pricing  data  embraces  more  than  historical  accounting  data.  It  also 
includes  such  factors  as  all  vendor  quotations,  nonrecurring  costs,  changes 
in  production  methods  and  production  or  procurement  volune,  data  in  support 
of  contractor  projections  of  business  prospects  and  objectives,  together 
with  related  costs  of  operations,  unit  cost  trends  such  as  those  associated 
with  labor  efficiency,  make  or  buy  decisions  and  estimated  resources  to 
attain  business  goals  and  any  other  management  decisions  which  reasonably 
could  be  expected  to  have  a  significant  bearing  on  costs  under  a  proposed 
contract,  e.g.,  the  comparative  analysis  by  which  a  particular  vendor  was 
selected.  In  short,  cost  or  pricing  data  consist  of  all  facts  which 

reasonably  can  be  expected  to  contribute  to  sound  estimates  of  future  costs 

as  well  as  to  the  validity  of  costs  already  incurred.  Cost  or  pricing 

data,  being  factual  ,  are  that  type  of  information  which  can  be  verified. 

(2)  Price  analysis  is  the  process  of  examining  and  evaluating  a 
prospective  price  without  evaluation  of  the  separate  cost  elements  and 
profit  proposed  by  the  individual  prospective  supplier  whose  price  is  being 
eval  uated . 

(3)  Cost  analysis  is  the  review  and  evaluation  of  a  contractor's 
cost  or  pricing  data  and  of  the  judgmental  factors  applied  in  projecting 
from  the  data  to  the  estimated  costs,  in  order  to  form  an  opinion  on  the 
degree  to  which  the  contractor's  proposed  costs  represent  what  performance 
of  the  contract  should  cost,  assuming  reasonable  economy  and  efficiency. 

(d)  Requirements  for  Price  or  Cost  Analysis.  Some  form  of  price  or 
cost  analysis  is  required  in  connection  with  every  negotiated  procurement 
action.  The  method  and  degree  of  analysis,  however,  is  dependent  on  the 
facts  surrounding  the  particular  procurement  and  pricing  situation.  Cost 

analysis  shall  be  performed  when  cost  or  pricing  data  are  required  to  be 

submitted.  The  extent  of  cost  analysis  should  be  that  necessary  to  assure 
reasonableness  of  the  pricing  result,  taking  into  consideration  the  amount 
and  complexity  of  the  proposed  contract.  Price  analysis  should  be  used  in 
all  other  instances  to  determine  the  reasonableness  of  the  proposed 
contract  price.  Normally,  a  sound  conclusion  as  to  value  cannot  be  made  on 
cost  analysis  alone.  Depending  on  the  information  available,  a  price 
arrived  at  by  cost  analysis  should  be  corroborated  through  price  analysis 
techni ques . 

3-807.2  Price  Analysis  and  Cost  Analysis  Techniques. 

(a)  Price  analysis  may  be  accomplished  in  various  ways  including: 

(1)  the  comparison  of  the  price  quotations  submitted; 

(2)  the  comparison  of  prior  quotations  and  contract  prices  with 

current  quotations  for  the  same  or  similar  end-items  (to  provide  a  suitable 
basis  tor  comparison,  appreciate  allowances  must  be  made  for  differences 
in  such  factors  as  time  of  prior  purchases,  specifications,  quantities 
ordered,  time  for  delivery,  Government-furnished  materials,  and  experienced 
trends  of  improvement  in  production  efficiency;  it  must  also  be  recognized 
that  such  comparison  may  not  detect  an  unreasonable  current  quotation 
unless  the  reasonableness  of  the  prior  prices  was  established  and  unless 
changes  in  the  general  level  of  business  and  have  been  considered); 

(3)  the  use  of  parametric  relationship  measurements  or  rough 
yardsticks  (such  as  dollars  per  pound,  per  horsepower,  or  other  units),  to 
point  up  apparent  gross  inconsistencies  which  should  be  subjected  to 
greater  pricing  inquiry; 
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(4)  trie  comparison  of  prices  set  forth  in  published  price  lists 
issued  on  a  competitive  basis,  published  market  prices  of  commodities  and 
similar  indicia,  together  with  discount  or  rebate  arrangements ;  and 

(5)  the  comparison  of  proposed  prices  with  estimates  of  cost 
independently  developed  by  the  Government. 

(b)  (1)  Cost  Analysis  includes  tne  appropriate  verification  of  cost  or 
pricing  data,  tne  evaluation  of  specific  elements  of  costs  and  the 
projection  of  tnese  data  to  determine  the  effect  on  prices  of  such  factors 
as : 

(i)  the  necessity  for  certain  costs; 

(ii)  the  reasonabl eness  of  amounts  estimated  for  tne  necessary 

costs ; 

(iii)  allowances  for  contingencies ; 

(iv)  tne  basis  used  for  allocation  of  indirect  costs;  and 

(v)  the  appropriateness  of  allocations  of  particular  indirect 
costs  to  the  proposed  contract. 

(2)  Cost  analysis  snail  include  appropriate  verification  that  the 
contractor's  cost  submissions  are  in  accordance  with  Section  XV,  Contract 
Cost  Principles  and  Procedures,  and  when  applicable  Appendix  U,  Rules, 
Regulations  and  Standards  of  the  Cost  Accounting  Standards  Board. 

(3)  Among  the  evaluations  tnat  snouid  be  made,  where  tne  necessary 
data  are  available,  are  comparisons  of  a  contractor’s  or  offeror's  current 
estimated  costs  with: 

(i)  actual  costs  previously  incurred  by  the  contractor  or 

of  feror ; 

(ii)  either  nis  last  prior  cost  estimate  or  a  series  of  prior 
estimates  for  tne  same  or  similar  items; 

(iii)  current  cost  estimates  from  otner  possible  sources; 

(iv)  prior  estimates  or  historical  costs  of  other  contractors 
manufacturing  the  same  or  similar  items;  and 

(v)  forecasts  or  planned  expenditures. 

(4)  Forecasting  future  trends  from  historical  cost  experience  is  of 
importance,  but  care  must  be  taken  to  assure  that  the  effect  of  past 
inefficient  or  uneconomical  practices  are  not  projected  into  the  future. 

An  adequate  cost  analysis  must  include  an  evaluation  of  trends  and  changes 
in  ci rcumstances ,  if  any,  and  their  effect  on  future  costs. 

3-807.3  Requirements  for  Cost  or  Pricing  Data 

(a)  When  appropriate,  the  contracting  officer  snail  require  the 
contractor  to  submit,  either  actually  or  by  specific  identification  in 
writing,  cost  or  pricing  data  in  support  of  his  proposal.  The  contracting 
officer  also  shall  require  tne  contractor,  in  circumstances  specified  in 
(b)  below,  to  certify,  using  the  certificate  set  forth  in  3-807.6,  that  the 
cost  or  pricing  data  are  accurate,  complete,  and  current.... 

(b)  Cost  or  pricing  data  are  required  as  part  of  a  proposal  leading 
to,  and  certification  is  required  prior  to: 

(i)  the  award  of  any  negotiated  contract  (except  for  unpriced 
actions  such  as  letter  contracts)  expected  to  exceed  $100,000  in  amount.... 

(iii)  ...unless  the  price  negotiated  is  based  on  adequate  price 
competition,  established  catalog  or  market  prices  of  commercial  items  sold 


In  substantial  quantities  to  tne  general  public,  or  prices  set  by  law  or 
regulation.... 

(c)  Unless  required  to  be  submitted  on  one  ot  tne  termination  forms 
set  forth  in  16-700,  data  will  be  submitted  on  the  DD  Form  633.  The 
requirement  for  submission  of  cost  or  pricing  data  is  met  when  all  cost  or 
pricing  data  reasonably  available  to  the  contractor  nave  been  submitted  or 
identified  in  writing  at  tne  time  ot  agreement  on  price.  Tne  data  snail  be 
submitted  to  the  contracting  officer  or  his  representative.  There  is  a 
clear  distinction  to  be  made  between  “submitting"  cost  or  pricing  data  and 
merely  "making  available"  books,  records  and  other  documents  without 
identification.  The  latter  does  not  constitute  submission  of  cost  or 
pricing  data. 

3-8U7.4  Subcontractor  Cost  or  Pricing  Data. 

(a)  Any  contractor  who  is  required  to  submit  and  certify  cost  or 
pricing  data  shall  be  required  to  submit  accurate,  complete  and  current 
cost  or  pricing  data  from  prospective  subcontractors  in  support  ot  each 
subcontract  cost  estimate  included  in  the  contractor's  submission.  This 
submission  shall  be  required  whenever  the  contracting  officer  considers 
such  data  necessary  for  pricing  tne  prime  contract,  or,  in  any  event, 
whenever  such  subcontractor  cost  estimate  is  either  (i)  $1,000,000  or  more, 
or  (ii)  both  more  tnan  $100,000  and  more  than  10  percent  of  the  prime 
contractor's  proposed  contract  price.... 

(b)  ...Failure  by  the  contractor  to  submit  subcontract  cost  or  pricing 
data  may  be  the  cause  for  disqual ification  of  the  contractor  from  further 
consideration  for  award  of  the  proposed  contract.... 

-  3-809  Contact  Audit  as  a  Pricing  Aid. 

(a)  General .  Contract  audit  services  are  available  in  two  forms: 

(i)  Audit  reports  setting  forth  tne  results  of  auditor's  reviews  and 
analyses  ot  cost  data  submitted  by  contractors  as  part  ot  pricing 
proposals,  reviews  ot  contractor's  accounting  systems,  estimating  methods, 
and  other  related  matters;  and 

(ii)  "On  the  spot"  personal  consultation  and  advice  to  contracting  and 
contract  administration  personnel  in  connection  with  analyses  of 
contractors'  cost  representations  and  related  matters  by  liaison  auditors 
stationed  at  contracting  and  contract  administration  offices.... 

(b)  Audit  Reports  on  Contractor  Price  Proposals.... 

(c)  Additional  Functions  ot  the  Contractor  Auditor.... 


PAR  Section  IV:  Special  Types  and  Methods  of  Procurement 


-  4-106.2  Solicitation. 

(f)  Solicitations  shall  require  offerors  to  identify  technical 
uncertainties  and  to  make  specific  proposals  for  their  resolution. 

(g)  Solicitation  and  evaluation  of  proposals  should  be  planned  to 
minimize  offerors'  and  Government  expense. 

-  4-106.4  Evaluation  for  Award. 

(a)  Generally,  research  and  development  contracts  should  be  awarded  to 
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tnose  organizations,  including  educational  organizations,  which  have  tne 
nignest  competence  in  tne  specific  field  of  science  or  tecnnolog y  involved. 

(c)  in  determining  to  wnom  tne  contract  snail  be  awarded,  the 
Contracting  Officer  snail  consider  not  only  technical  competence,  Dut  also 
all  other  pertinent  factors  including  management  capabil ities ,  cost 
controls,  and  past  performance  in  adhering  to  contract  requirements, 
weigning  eacn  factor  in  accordance  witn  the  requirements  of  tne  particular 
procurement....  Proposals  for  cost  reimbursement  type  or  fixed  price 
incentive  contracts  may  be  penalized  during  evaluation  to  tne  degree  that 
tne  estimated  cost  is  unrealistically  low. 

-  4-106.5  Evaluation  of  Price  and  Costs. 

(a)  kinile  cost  or  price  snould  not  be  tne  controlling  factor  in 
selecting  a  contractor  for  a  researcn  or  development  contract,  cost  or 
price  snould  not  De  disregarded  in  tne  choice  of  tne  contractor.  It  is 
important  to  evaluate  a  proposed  contractor's  cost  or  price  estimate,  not 
only  to  determine  whether  tne  estimate  is  reasonable  but  also  to  determine 
ms  understanding  of  tne  project  and  ability  to  organize  and  perform  tne 
contract.  Tne  most  useful  tools  for  tnis  purpose  are  price  analysis  and 
cost  analysis  (see  3-807. 2). 

(d)  Price  analysis  generally  considers  the  overall  reasonableness  of 
tne  proposals  in  relation  to  tne  total  contemplated  expenditures  and  the 
extent  and  nature  of  tne  task  scneduled  to  be  accompl isned.  In  most 
research  and  development  contracts,  tne  inability  to  define  specifications 
and  tne  nature  of  tne  end  items  prevent  the  effective  use  of  certain 
tecnniques  of  price  analysis,  sucn  as  comparisons  with  prior  quotations  and 
current  prices  evaluations  in  terms  of  quantitative  yardsticks.  Tne 
conclusions  (eacned  oy  price  analysis  techniques  must  be  supported  by  cost 
analysis  procedures,  used  to  examine  tne  details  of  tne  offerors' 
proposal s. 

(c)  Tne  analysis  of  cost  factors  begins  witn  an  evaluation  of  the 
reliability  of  the  offeror's  cost  estimating  procedures  and  the 
dependability  of  ms  cost  controls,  as  demonstrated  by  his  history  of  cost 
management  in  the  performance  of  otner  contracts  or  by  his  establishment  of 
sound  practices  for  tnis  purpose.  Tne  cost  analysis  proceeds  witn  a 
critical  examination  of  the  composition  of  eacn  cost  element  in  terms  of 
its  expected  application  to  tne  objectives  of  tne  contract,  and  its 
conformance  to  tne  accepted  principles  of  allocability  and  reasonableness. 
(See  Section  XV,  Part  3,  and  1 5-201 . )  A  Government  cost  estimate  may  nelp 
in  projecting  tools  for  tnese  purposes  and  may  develop  the  expected 
incidence  of  various  cost  factors  in  relation  to  performance  pnases, 
planned  segments,  or  identifiable  “milestones."  Tnis  estimate  snould 
provide  a  summary  forecast  of  tne  time,  effort,  materials,  equipment,  and 
services  necessary  to  accompl isn  tne  researcn  or  development  objective. 

Ine  comparison  and  reconciliation  of  tne  Government  cost  estimate  witn  tne 
offeror's  cost  estimate  for  tne  same  phases,  segments,  or  events  snould 
provide  a  foundation  for  meaningful  discussions  witn  the  offeror. 

(d)  bpecial  care  snould  be  exercised  to  comply  witn  15-205.1  and 
15-205.33  in  tne  allowance  of  advertising  costs  under  15-309.1. 
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DAK  Section  XV:  Contract  Cost  Primples  and  Procedures 


-  15-000  Scope  ot  Section.  ...contains  general  cost  principles  and 
procedures  for  tne  pricing  of  contracts  and  contract  modifications  whenever 
cost  analysis  is  performed  (see  3-807.2),  and  tor  the  determination, 
negotiation,  or  allowance  ot  costs  when  such  action  is  required  by  a 
contract  clause. 

-  15-109  Definitions.... 

(a)  Profit  Center.... 


(m)  Reporting  Costs 

-  Part  2-  Contracts  with  Commercial  Organizations 


-  15-201.1  Composition  of  Total  Cost.  The  total  cost  ot 
sum  of  tne  allowable  direct  and  indirect  costs  allocable 
incurred  or  to  be  incurred,  less  any  allocable  credits, 
what  constitutes  costs,  any  generally  accepted  method  ot  determining  or 
estimating  costs  that  is  equitable  under  the  circumstances  may  be  used, 
including  standard  costs  properly  adjusted  for  applicable  variances. 


a  contract  is  the 
to  the  contract. 
In  ascertaining 


-  15-201.2  Factors  Affecting  Allowability  of  Costs.  Factors  to  be 
considered  in  determining  the  allowability  ot  individual  items  of  cost 
include  (i)  reasonableness,  (ii)  al locabi 1 ity. . . . 

-  15-201.3  Definition  of  Reasonableness. 

(a)  General.  A  cost  is  reasonable  if,  in  its  nature  or  amount,  it  does 
not  exceed  that  which  would  be  incurred  by  an  ordinarily  prudent  person  in 
the  conduct  of  competitive  business.  The  question  ot  the  reasonableness  of 
specific  costs  must  be  scrutinized  with  particular  care  in  connection  with 
firms  or  separate  divisions  thereof  which  may  not  be  subject  to  effective 
competitive  restraints.  What  is  reasonable  depends  on  a  variety  of 
considerations  and  circumstances  involving  both  the  nature  and  the  amount 
of  the  cost  in  question.  In  determining  the  reasonableness  of  a  given 
cost,  consideration  shall  be  given  to- 

(i)  whether  the  cost  is  of  a  type  generally  recognized  as  ordinary  and 
necessary  for  the  conduct  of  the  contractor's  business  or  the  performance 
of  the  contract; 

(ii)  the  restraints  or  requirements  imposed  by  such  factors  as 
generally  accepted  sound  business  practices,  arms  length  bargaining, 

Federal  and  State  laws  and  regulations,  and  contract  terms  and 
specifications; 

(iii)  the  action  that  a  prudent  businessman  would  take  in  the 
circumstances,  considering  his  responsibilities  to  the  owners  ot  the 
business,  his  employees,  his  customers,  the  Government  and  tne  public  at 
large;  and 

(iv)  significant  deviations  from  the  established  practices  ot  the 
contractor  which  may  unjustifiably  increase  the  contract  costs. 
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NAVMAT  Instruction  4200.49 

Selection  of  Contractual  Sources  for  Major  Defense  Systems 
28  February  1977 

Excerpts  related  to  cost  evaluation: 

-  Instruction  amplifies  DOD  Directive  4105.62,  same  title,  where 
appropriate. 

-  3.  The  provisions  apply  to  (i)  competitive  procurements  covering  one  or 
more  contractors  as  a  prime  development  and/or  production  contractor( s)  for 
programs  which  have  an  estimated  RDT&E  cost  in  excess  of  $75  mil  1  ion .. .and 
(ii)  such  other  competitive  procurement  as  may  be  so  directed  by  the  Chief 
of  Naval  Material  . 

-  4.b.  Unless  otherwise  specified  by  the  Secretary  of  Defense  for  a 
specific  program,  the  Secretary  of  the  Navy  (SECNAV)  is  the  Source 
Selection  Authority  (SSA)  for  all  Navy  defense  systems  designated  major 
pursuant  to  reference  (a),  with  power  of  delegation  at  his  discretion, 
(reference  (a)  is  DOD  Directive  5000.1). 

-  5. a.  Unless  otherwise  approved  by  the  Source  Selection  Authority,  each 
program  subject  to  the  provisions  of  this  instruction  shall  establish  and 
effectively  employ  an  evaluation  group  structure  consisting  of  a  Source 
Selection  Advisory  Council  (SSAC)  and  Source  Selection  Evaluation  Board 
(SSEB)  except  that  such  a  structure  is  not  mandatory  where  the  source 
selection  decision  is  to  be  based  solely  on  price. 

-  6.  The  Source  Selection  Plan.  The  Source  Selection  Plan  (SSP)  shall 
describe  in  detail  the  plan  for  the  conduct  of  the  entire  source  selection 
process  including  solicitation  preparation  and  review,  proposal  evaluation, 
source  selection  and  contract  award.  As  a  minimum,  the  SSP  shall 
contain. ,.(iv)  the  evaluation  criteria  listed  in  the  relative  order  of 
importance;  (v)  the  evaluation  procedures  to  be  employed  by  the  evaluators 
including  the  methods  for  scoring/grading  the  proposals;  but  excluding  any 
numerical  weights  which  may  be  employed.... 

-  7.  Preparation  of  the  Solicitation. 

a.  The  mere  listing  of  the  evaluation  criteria  in  relative  order  of 
importance  may  not  suffice  to  even  inform  prospective  offerors  of  the  broad 
basis  on  which  their  proposals  will  be  evaluated.  For  example,  there  may 
be  a  situation  where  the  importance  of  a  single  evaluation  factor  far 
outweighs  all  other  evaluation  factors.  When  such  a  situation  exists,  the 
predominant  value  accorded  this  factor  should  be  disclosed  along  with  the 
relative  order  of  importance  of  the  remaining  factors  provided,  however, 
that  the  ASPR  3-501  Section  d(i)  prohibition  against  the  disclosure  of  any 
numerical  weights  which  may  be  employed  in  the  evaluation  process  is  not 
viol ated . 

-  Enclosure  (2)  Duties  and  Resposibil ities  of  the  Source  Selection 
Authority  (SSA). 
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5.  Ensure  that  cost  {devel opment ,  production,  and  support)  is 
appropriately  considered  as  an  integral  element  of  proposal  evaluation, 
together  with  technical,  performance,  schedule  and  logistic  support 
objectives,  in  accordance  with  the  relative  order  of  importance  of  the 
evaluation  and  selection  criteria. 

-  Enclosure  (4)  Duties  and  Responsibilities  of  the  Procuring  Contracting 
Officer . 


S.  Assure  that  proposals  are  requested  on  a  basis  which  permits 
verification  of  their  cost  credibility. 

-  Enclosure  (5)  Duties  and  Responsibilities  of  the  Source  Selection 
Advisory  Council  . 

2.  Develop  the  criteria  for  evaluation  and  selection  in  a  relative 
order  of  importance  and  establish  any  numerical  weights  which  may  be 
an  ployed . 

-  Enclosure  (7)  is  a  source  selection  schematic. 
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NAVELEX  Instruction  4200. 12C 

Proposal  Evaluation  and  Source  Selection 

31  August  1978  with  Change  1  of  5  February  1979 

Excerpts  related  to  cost  evaluation: 

-  3.  Applicability.  The  provisions  of  this  instruction  are  applicable  to 
all  competitive  negotiated  procurements.... 

-  5.  Policy.  It  is  the  policy  of  the  Naval  Electronics  Systems  Command  to 
assure  impartial  equitable  and  comprehensive  evaluation  of  offers  and 
selection  of  the  source  or  sources  whose  offer  provides  the  greatest 
advantage  to  the  Government  after  consideration  of  performance,  schedule 
and  cost. 

-  6.  Thresholds. 

a.  Major  Defense  System.  Program  procurements  exceed ing...RDT&E  cost 
in  excess  of  75  million  dollars  or  production  costs  in  excess  of  300 
million  dollars  will  in  all  cases  employ  the  formal  source  selection  method 
prescribed  in  Enclosure  (1). 

b.  Other  Procurements....  Generally,  the  formal  selection  procedures 
shall  not  be  considered  for  procurements  under  10  million  dollars,  except 
under  the  most  exceptional  circumstances.... 

-  7.  Solicitation  Provisions. 

a.  General....  Evaluation  factors  will  be  limited  to  those  which  are 
essential  to  the  selection  of  the  offeror  most  likely  to  satisfy  the 
Government's  requirements  in  terms  of  cost,  schedule  and  performance. 
Generally,  offerors  shall  be  required  to  submit  cost/price  information 
separately  from  technical  portions  of  proposals.... 

-  10.  Timeliness  of  the  Source  Selection  Process.  Source  selection  must  be 
completed  expeditiously....  Under  normal  ci rcumstances ,  the  evaluation 
period  should  not  exceed  thirty  days.... 

-  13.  Basis  for  Proposal  Evaluation. 

a.  Evaluation.  Contracts  will  be  awarded  on  the  basis  of  maximum 
advantage  to  the  Government,  cost  or  price  and  other  factors  considered.... 
A  numerical  scoring  system  will  be  employed  which  will  translate  the  word 
descriptions  into  quantitative  terms.. ..(see  Enclosure  (3)). 

b.  Weighting.  All  evaluation  factors  listed  in  the  RFPwill  be 
assigned  weights,  except  that  weighting  cost  is  not  mandatory  for  proposed 
cost  type  contracts  for  early  conceptual  effort  involving  high  technical 
risk.  Cost  may  be  excluded  from  weighting  in  such  cases  with  the 
concurrence  of  the  PC0;  however  the  evaluation  of  costs  will  precede  and  be 
a  factor  in  the  establishment  of  competitive  range  and  the  relationship  of 
cost  factors  to  other  evaluation  criteria  will  be  adjudicated  to  the 
satisfaction  of  the  PC0  in  recommendation  and  selection  for  award.  The 
summation  of  weighted  scores  will  not  be  the  sole  basis  for  the  selection 
of  a  source.  Such  scores,  however,  may  provide  compelling  support  for 
determinations  of  competitive  range  or  source  selection.  For  that  reason, 
it  is  imperative  that  weighted  scores  reflect  as  accurately  as  possible  the 


substantive  aspects  of  the  proposals  and  their  differences,  and  that  such 
scores  positively  discriminate  between  proposals  of  significantly  varying 
merit,  where  such  is  the  case....  Every  effort  should  be  made  to  insure 
that  the  evaluation  process  is  not  reduced  to  a  mere  statistical 
computation.  These  evaluation  weights  will  be  kept  by  the  Chairman  of  the 
CARP  and  will  normally  be  available  only  to  the  SSA  and  the  PCO. 

c.  Evaluation  Results.  Generally,  evaluation  results  should 
categorize  proposals  as  (1)  acceptable  (i.e.,  acceptable  for 
discussion/negotiation  purposes),  (2)  susceptible  of  being  made  acceptable 
(i.e.,  where  deficiencies  are  such  that  a  reasonable  amount  of  discussion 
may  make  the  proposal  acceptable)  and  (3)  unacceptable  (i.e.,  where  such 
deficiencies  exist  as  to  preclude  possiblitiy  of  meaningful  discussions  and 
opportunity  for  award).  In  the  event  that  a  proposal  contains  a  major 
deficiency  in  a  critical  area,  the  proposal  may  be  determined  to  be 
unacceptable  despite  an  overall  numerical  score  supporting  an  acceptable 
rating,  provided,  however,  that  such  criteria  are  indentified  in  the 
solicitation  and  are  incorporated  in  the  evaluation  plan. 

-  15.  Competitive  Range. 

a.  Basis  for  Determination....  A  proposal  is  within  the  competitive 
range  unless  so  technically  inferior  or  out  of  line  in  price  that 
meaningful  negotiations  are  precluded.  Cost  or  price  as  well  as  technical 
factors  must  be  considered  in  determination  of  the  competitive  range.... 

-  Enclosure  (1):  Formal  Source  Selection  Procedures. 

3.  Source  Selection  Plan. 

a.  Preparation  and  Approval....  The  plan  shall  include  as  a 
minimum.... 

b.  Evaluation  of  Costs.  To  assist  the  SSAC  in  evaluating  the  cost  or 
price  proposals,  the  Source  Selection  Plan  may  provide  for  a  special  panel 
similar  to  the  SSEB  to  evaluate  the  proposals  and  rank  them  according  to 
their  overall  costs  or  prices  as  well  as  cost  credibility,  if  appropriate. 
This  special  panel's  report  shall  be  used  in  conjunction  with  the  SSEB 
report  to  form  the  basis  for  the  SSAC's  recommendations.  The 
PCO/negotiator  will  normally  be  a  member  of  the  cost  evaluation  panel. 

-  Enclosure  (2):  Standard  Procedures  for  Proposal  Evaluation  and  Source 
Selection 

1.  General....  The  documentation  developed  during  the  evaluation 
process  will  be  adequate  to  both  support  the  action  taken  and  to  debrief 
unsuccessful  offerors. 

-  Enclosure  (3):  Scoring  Techniques  and  Standards 

1.0  Detailed  evaluation  of  proposals  will  be  made  by  individual 
evaluators  who  will  consider  the  merits  of  specific  portions  of  each 
proposal  against  the  established  criteria.  Primary  emphasis  in  evaluating 
proposals  will  be  placed  on  the  analysis  of  the  way  each  offeror  proposes 
to  satisfy  the  RFP  requirements,  the  risks  and  costs  associated  with  the 
proposed  effort,  and  the  reduction  of  such  analysis  to  descriptive  words 
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and  phrases.  The  numerical  scoring  system  detailed  herewith  will  be  used 
to  translate  the  word  descriptions  into  quantitative  terms....  The 
objectives  of  using  these  standards  are  to  promote  uniformity  in  evaluation 
and  to  aid  in  comparison  and  understanding  of  evaluation  results. 

The  following  standards  are  established  for  this  evaluation: 

Excellent,  good,  fair,  poor,  and  unacceptable. 
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NAVA IR  Instruction  4200.24 

Selection  of  Contractual  Sources  for  Major  Aircraft  and  Missile  System 
Acqui  sitions 
1  July  1977. 

Excerpts  related  to  cost  evaluation: 

-  Implements  NAVMATINST  4200.49  of  28  Feb  1977 

-  3.  Background.  ...It  is  the  objective  of  the  proposal  evaluation  and 
source  selection  process  to  select  the  source/design  which  best  satisfies 
the  requirements  of  the  Government  at  an  affordable  cost. 

-  4.  Policy  and  General  Procedures.  It  is  the  policy  of  NAVAIR  HQ  to 
utilize  design  competitions  for  selecting  contractors  for  the  development, 
design  and  manufacture  of  major  weapon  systems.  In  carrying  out  this 
policy,  the  following  will  be  accomplished: 

a.  Effect  source  selection  procedures  which  will  choose  the  best 
weapon  system  design  from  those  potentially  available,  and  which  result  in 
the  greatest  gain  in  military  effectiveness  (as  generally  established  by 
the  applicable  operational  requirement)  within  the  expected  available 
resources  (time,  funds,  and  facilities)  and  other  known  constraints. 

b.  Effect  all  source  selections  with  rigorous  impartiality. 

c.  Utilize  procedures  which,  in  themselves,  make  effective  and 
efficient  use  of  time,  money  and  manpower  by  both  the  Government  and  the 
offerors . 

d.  Establish  as  explicitly  as  practical  all  known  constraints  which 
must  be  satisfied  by  the  weapon  system  to  be  selected.  These  constraints 
shall  include  (1)  time...,  (2)  total  and  incremental  cost  constraints,  (3) 
operational  constraints ..  .and  (4)  policy  and  other  constraints. 

-  5.  Organization. 

a.  Source  Selection  Authority(SSA) .  The  duties  and  responsibilities 
of  the  SSA  are  contained  in  NAVMATINST  4200.49. 

b.  Source  Selection  Advisory  Council  (SSAC)....  In  addition  to  the 
responsibilities  set  forth  in  reference  (a),  the  SSAC  shall  also  perform 
the  following: 

(1).... 

(5)  Assure  that  the  source  selection  procedures  promote  consistency  in 
an  integrated  evaluation  of: 

(a)  estimated  military  effectiveness  of  the  weapon  system; 

(b)  estimated  life  cycle  cost  of  the  weapon  system;  and 

(c)  risk  associated  with  each  of  the  above  two  estimates. 

c.  Source  Selection  Evaluation  Board  (SSEB)....  The  SSE8  shall  also 
perform  the  fol  lowing :...  (2)  Eval  uate  the  proposals  against  the  technical 
requirements  and  military  effectiveness  models  and  estimate :.. .(c)  the  life 
cycle  cost  of  the  weapon  system. 

d.  Source  Selection  Plan  (SSP).  A  plan  for  conducting  the 
solicitation,  proposal  evaluation,  negotiation,  and  source  selection. 
Matters  to  be  described  in,  and  connected  with,  the  SSP  are  covered  in 
reference  (a).  (Reference  (a)  is  NAVMATINST  4200.49.) 
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-  6.  General  Source  Selection  Process 

a.  Announcement  of  Solicitation.... 

b.  Solicitation  (RFP/RFQ).... 

c.  Bidder's  Conference.... 

d.  Evaluation  Procedure.  The  evaluation  of  proposals  and  quotations 
shall  be  carried  out  against  standards  established  in  the  SSP  and  the 
RFP/RFQ.  Ad  hoc  groups  may  be  established  within  the  various  disciplines 
(such  as  technical  ,  logistic,  cost,  management)  to  support  the  SSEB,  if 
desired....  It  is  essential  that  each  evaluator  have  the  necessary 
expertise  to  perform  a  valid  analysis  by  independent  and  realistic 
appraisal.  These  skills  must  be  built  up  in  a  core  of  knowledgeable  and 
professionally  competent  personnel  in  specialty  areas  from  past  experience 
in  source  selection  and  in  similar  or  related  programs  to  provide  essential 
mature  judgment  and  expertise  in  the  evaluation.... 

e.  Inquiries  to  Offerors.  During  the  evaluation,  it  may  be  desirable 
to  make  inquiries  to  the  offerors  to  eliminate  uncertainties  or 

i rregul ari t ies  and/or  provide  the  offeror  an  opportunity  to  better 
substantiate  a  proposed  technical  approach  or  solution.  This  would 
normally  occur  when  such  inquiries  would  provide  for  more  effective 
competition.  All  inquiries  will  be  made  through,  or  in  the  presence  of, 
the  Contracting  Officer  or  his  designated  representative.  Deficiencies 
clearly  related  to  an  offeror's  judgment,  or  his  lack  of  competence  or 
inventiveness  in  preparing  his  proposal,  shall  not  be  disclosed. 

f.  Competitive  Range  and  Discussions  with  Offerors  (ASPR  3-805.2  and 
3-805. 3) . . . . 

g.  Final  Negotiations.  Negotiations  with  otferor(s)  will  be  conducted 
by  the  Contracting  Officer  to: 

(3)  assure  the  reasonableness  of  the  proposed  costs/prices. 

h .  Eval  uation  Report .. . . 

i.  Award.  Based  on  facts  and  findings  of  the  SSEB  and  the 
recommendations  of  the  SSAC,  the  SSA  will  make  the  decision  as  to  source 
sel ection( s) .  Award  will  be  on  the  basis  of  the  overall  bes+  oroposal  to 
the  Navy,  technical,  cost,  and  management  factors  considered.... 
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NAVAIR  Instruction  4200.27 

Selection  of  Contractual  Sources  for  Less-Than-Major  Competitive 
Acquisitions 
7  November  1980 

Excerpts  related  to  cost  evaluation: 

-  5.  Definitions.... 

b.  Less-Than-Major  Acquisition.  Those  competitive  procurements  that 
do  not  meet  the  cost  criteria  established  for  major  acquisitions  or  are  not 
designated  as  major  acquisitions  by  higher  authority. 

-  6.  Policy. 

a.  ...Formal  source  selections  shall  follow  the  procedures  defined  in 
enclosure  (1)....  Formal  source  selection  procedures  shall  be  used  for 
those  procurements  which  involve  technical  competition  if  the  acquisition 
cost  is  greater  than  $5.0M  RDT&E  or  $25. 0M  production....  Technical 
competition  exists  when  at  least  50  percent  of  the  evaluation  criteria  for 
award  is  based  on  technical  or  other  factors  in  addition  to  price  or 
cost .... 

-  Enclosure  (1):  Formal  Source  Selection  Procedures  and  Requirements 

3c.  ...the  SSE8  shall  al so...eval uate  the  proposals  against  the 
solicitation  requirements  and  estimate.. ,{c)  the  life  cycle  cost  associated 
with  the  proposed  item  or  concept  design.... 

5.  Source  Selection  Criteria....  The  criteria  must  include  all 
relevant  areas  (for  example,  technical,  cost,  supportabil  ity ,  schedule, 
risk  and  management)  in  the  relative  order  of  importance.  They  must  be 
stated  with  sufficient  generality  to  encompass  all  evaluation  issues, 
including  ones  which  may  not  be  foreseen  at  the  time  of  SSP  preparation. 
Lengthy,  detailed  criteria  which  attempt  individually  to  address  all 
evaluation  elements  shall  be  avoided. 

9b.  It  is  often  argued  that  rating  of  proposals  by  the  SSEB  should  be 
accomplished  via  detailed,  pre-established  numerical  scoring  systems. 
However,  experience  has  shown  that  use  of  such  systems  is  contrary  to  the 
goal  of  an  efficient  source  selection  and,  moreover,  that  such  systems 
often  lead  to  a  selection  which  is  other  than  the  one  most  advantageous  to 
the  Government.  Therefore,  the  use  of  numerical  scoring  ("point")  systems 
by  the  SSEB  for  the  rating  of  proposals  is  strongly  discouraged.  The  use 
of  pre-establ ished  point  systems  is  prohibited.  A  pre-estabished  point 
system  is  one  that  is  formulated  and  defined  prior  to  receipt  of  the 
proposals.  It  represents  an  attempt  by  members  of  the  SSEB  to  prejudge  the 
relative  absolute  importance  of  the  many  unidentified  subelements  of  the 
selection  criteria  prior  to  knowing  what  all  of  those  subelements  are  or 
exactly  how  they  relate.  Instead,  quantitative  factors  (for  example, 
speed,  range,  cost)  shall  be  measured  in  their  normal  units  of  measure 
(that  is,  knots,  nautical  miles,  dollars)  and  compared  against  the  RFQ/P 
requirements,  if  established,  and  qualitative  factors  shall  be  treated 
using  descriptive  terms  such  as  those  shown  in  Table  IV.  Such  ratings 
always  must  be  supported  by  specific  rationale  developed  by  the  evaluation 
process. 
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13.  ...Discussions/negotiations  will  be  held  with  the  offerors  who  are 
within  the  competitive  range.  Contracting  Officer  negotiations  with  such 
offerors  shal 1 : 

(1)  resolve  all  exceptions  made  by  NAVAIR  HQ  and  the  offerors  to  the 
terms  and  conditions  of  proposed  contracts; 

(2)  incorporate  technical  and  specification  changes  determined  by  the 
SSE8  to  be  necessary; 

(3)  advise  the  offerors  in  those  areas  where  their  proposal  does  not 
meet  solicitation  requirements;  and 

(4)  ensure  the  reasonableness  of  the  proposed  costs/prices. 

15.  Award....  Award  will  be  on  the  basis  of  the  overall  best  proposal 
to  the  Navy,  technical  ,  cost,  schedule,  supportabil ity,  risk  and  management 
factors  considered....  Unsuccessful  offerors  shall  be  given  the 
opportunity,  if  they  so  request,  for  an  informal  de-briefing  on  the  results 
of  the  evaluation  of  their  particular  proposals.  It  is  of  benefit  to  the 
Government  and  to  such  offerors  to  identify  deficiencies  which  will  provide 
an  understanding  for  their  non-selection....  The  debriefings  shall  be 
limited  to  the  evaluation  of  important  aspects  of  the  proposal  against  the 
requirements  of  the  RFP/RFQ.... 

Sample  qualitative  proposal  ratings: 

Exceptional  ,  acceptable,  marginal  ,  and  unacceptable 
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AMC  Regulation  715-1 

Cost  Realism  in  Proposal  Evaluations 

1  October  1970 

Excerpts  related  to  cost  evaluation. 

-  1.  Purpose.  This  regulation  prescribes  the  objectives,  policy, 
responsibilities,  procedures,  and  guidance  in  the  preparation  and  use  of 
cost  realism  in  proposal  evaluation  procurements. 

-  3.  Definitions. 

a.  Cost  realism  is  defined  as  the  employment  of  preplanned  methods  to 
determine  the  probable  total  cost  for  a  procurement  at  completion;  cost 
realism  involves  a  comprehensive  analysis  to  develop  and  establish  the 
probable  overall  cost  of  performance  when  related  to  the  required  technical 
scope  of  the  work. 

-  4.  General.  Cost  realism  will  be  used  in  all  procurements  in  which 
multiple  evaluation  factors  are  set  forth  in  the  solicitation  document. 

-  5.  Objectives. 

a.  This  regulation  is  issued  to  stress  cost  realism  procedures  as  a 
method  used  to  reduce  cost  growth  and  inaccurate  cost  estimating  on  the 
part  of  both  the  Government  and  the  potential  contractors. 

b.  Cost  realism  procedures  will  be  used  during  the  presolicitation, 
solicitation,  cost  evaluation  and  negotiation  processes. 

-  6.  Pol  icy. 

a.  It  is  the  policy  of  Headquarters,  AMC,  that  Defense  contractors 
will  be  advised  of  the  need  for  cost  realism  in  their  proposals  and  that  it 
will  become  a  major  factor  in  the  selection  process  for  the  award  of 
contracts . 

b.  Cost  realism  procedures  will  be  used  in  consonance  with  sound  cost 
estimating  and  pricing  practices  to  determine  the  reasonableness  of 
prospective  contractors'  proposals. 

-  7.  Responsibil ities .. . . 

b.  The  Commander .. .wi 1 1  assure  that  qualified  pricing,  contracting, 
legal,  and  technical  personnel  are  available  to  assist  the  procuring 
contracting  officer  (PCO)  in  applying  cost  realism  procedures  as  prescribed 
by  paragraph  8. 

c.  Each  PCO  will  assure  that  cost  realism  procedures  are  used  during 
proposal  analysis  and  subsequent  negotiations  when  the  procurement  falls 
within  the  purview  of  this  regulation. 

-  8.  Procedures. 

a.  The  cost  realism  evaluation  plan  will  De  tailored  expressly  for 
each  procurement  and  the  method  outlined  in  the  solicitation  document. 
Examples  of  standards  which  may  be  used  for  evaluation  purposes  are: 

(1)  Are  the  proposed  elements  of  cost  (labor,  material,  and  other 
costs)  realistic,  reasonable,  and  consistent  with  the  work  to  be  performed, 
as  outlined  in  the  offeror's  technical  proposal? 
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(2)  Are  the  costs  so  excessively  high  or  excessively  low  as  to 
indicate  the  offeror's  lack  of  understanding  or  capability  to  perform? 

(3)  Does  the  cost  data  indicate  any  areas  of  inadequacy  or  areas  of 
nonessential  effort? 

(4)  How  does  the  contractor's  record  of  previous  cost  performances 
compare  with  the  initial  negotiated  amounts  on  past  contract  awards? 

(5)  How  well  has  the  contractor  associated  his  costs  to  the  technical 

scope  of  work  which  he  has  proposed? 

(6)  If  an  Independent  Government  Cost  Estimate  was  prepared,  how  well 
does  the  contractor's  proposal  compare  with  it? 

(7)  How  complete  is  the  cost  data  as  submitted  by  the  contractor? 

(8)  What  is  the  responsiveness  of  the  contractor  during  negotiations? 
Is  the  contractor  willing  to  accept  ceilings  on  particular  elements  of 
costs  such  as  overheads? 

b.  The  weight  assigned  to  cost  on  production  type  procurements  will 
not  be  less  than  30  percent  of  the  total  weight  to  be  used  for  evaluation 
purposes.  On  research  and  development  type  contracts,  an  appropriate 
weight  will  be  used  to  assure  that  cost  as  a  separate  element  will  be 
considered . 


Army  Regulation  No.  715-6 

Procurement  Proposal  Evaluation  and  Source  Selection 
21  September  1970 

Excerpts  related  to  cost  evaluation: 

-  3.  Objectives. 

a.  The  prime  objectives  of  proposal  evaluation  and  source  selection 
are  to — 

(1)  Insure  impartial,  equitable,  and  comprehensive  evaluation  of 
proposal  s. 

(2)  insure  selection  of  that  source  whose  proposal  ,  as  submitted, 
offers  optimum  satisfaction  of  the  Government's  objectives,  including  cost, 
schedule,  and  performance. 

-  5.  Principles  and  techniques. 

a.  Decision  latitude.  It  is  intended  that  the  source  selection 
authority  have  a  maximum  latitude  in  the  selection  decision.... 
e.  Techniques. 

(2)  When  numerical  evaluation  techniques  are  used,  summary  scores  for 
each  proposal  evaluated  should  be  the  mean  of  the  scores  of  each  evaluator 
assigned.  Prior  to  such  formal  individual  ratings,  in-depth  discussions 
among  the  evaluators  assigned  should  be  initiated  to  insure  understanding 
and  consideration  of  the  relative  merits  of  the  proposal  against  the 
established  technical  requirements  and  criteria. 

(3)  Numerical  scores  may  not  convey  fully  the  individual  evaluator's 
judgment  of  some  aspects  of  the  proposal;  therefore,  each  evaluator  should 
supplement  his  rating  with  concise  narrative  evaluation  which  includes 
discussion  and  interpretation  of  the  limitations  of  his  rating. 
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USAMC  Supplement  1  to  AR  715-6 

Procurement  Proposal  Evaluation  and  Source  Selection 
3  November  1970 

Excerpts  related  to  cost  evaluation: 

-  AR  715-6,  21  September  1970,  is  supplemented  as  follows: 

-  Page  1,  paragraph  3b.  Additional  specific  objectives  are: 

(5)  Insure  increased  weight  is  given  to  cost  realism  in  proposal 
evaluation  plans. 

(6)  Insure  offerors  are  made  aware  that  cost  realism  will  be  a  major 
factor  in  the  source  selection. 

-  Page  4,  paragraph  6c.  Responsibilities/Source  Selection  Advisory 
Council : 

(14)  Insure  that  records  of  offerors  past  performance  in  cost 
estimating  are  used  in  determining  the  realism  of  cost  estimates  in  current 
proposal  s. 

(15)  Insure  that  areas  of  contractor  proposals  that  lack  cost  realism 
are  thoroughly  reviewed  and  clarified  through  negotiations  by  the 
contracting  officer. 

(16)  Insure  that  the  SSEB  evaluates  the  results  of  negotiations  to 
clarify  lack  of  cost  realism  and  includes  such  evaluation  in  its  report. 

(17)  Insure  that  the  SSAC  analyses  highlight  the  results  of 
negotiation  to  resolve  the  lack  of  cost  realism. 
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OARCOM  Pamphlet  715-3 

Proposal  Evaluation  and  Source  Selection 

October  1980 

Excerpts  related  to  cost  evaluation: 

-  Foreward 

Although  the  provisions  of  this  pamphlet  are  not  mandatory,  they  are 
based  on  the  latest  experience  in  the  use  of  organized  proposal  evaluation 
and  source- sel ecti on  procedures  contemplated  by  AR  715-6,  Proposal 
Evaluation  and  Source  Selection.  Mandatory  instructions  for  proposal 
evaluation  and  source  selection  for  major  systems  are  contained  in  AR 
715-6. 

The  proposal  evaluation  and  suurce-sel  ection  process  is  dynamic  and 
requires  the  exercise  of  judgment  and  experience  throughout  the  process. 

As  acquisitions  vary  in  size  and  technical  complexity,  so  do  their 
requirements  for  proposal  evaluation  and  source- sel ection  procedures.... 

The  Source-Selection  Authority  must  choose  the  degree  of  application  of 
these  procedures,  drawing  on  the  advice  of  legal,  contracting,  and 
technical  personnel. 

-  Chapter  1.  Introduction 

1-1  Purpose....  Ultimately,  the  source  selection  shall  take  into 
account  the  offerors  capability  to  provide  the  Army  with  the  best  value. 

1- 3  Flexibil ity. 
a.  Tailoring. 

(1)  Due  to  the  variety  of  circumstances  that  may  be  encountered,  a 
static  organizational  structure,  timetable,  or  detailed  procedure  is  not 
practical  or  desirable. 

-  Chapter  2.  Organizational  Structure  and  Staffing  for  Proposal  Evaluation 
and  Source  Selection 

2- 2  Tailoring  an  organi zational  structure. 

a.  The  degree  of  technical  sophistication  as  well  as  the  dollar  value 
of  the  acquisition  are  among  those  considerations  that  normally  dictate  the 
magnitude  and  configuration  of  the  organization. 

b.  Every  acquisition  within  the  purview  of  this  pamphlet  goes  through 
the  same  functional  steps:  evaluation,  negotiation,  selection,  and  award. 

2- 3  Basic  organizational  model, 
b.  SSAC. 

(h)  Insure  that  contractor  perrormance  history  in  the  areas  of 
technical,  cost  realism,  and  management  is  evaluated  by  the  SSEB. 

-  Chapter  3.  Evaluation  and  Selection  Process 

3- 2c(ll)  Scoring  procedures. 

(a)  The  particular  scoring  system  to  be  used  in  the  evaluation  is 
described  in  this  portion  of  the  plan.  Of  the  several  types  available, 
including  numerical,  adjectival,  or  combination  thereof,  one  must  be  chosen 
that  is  tailored  to  the  requirements  of  the  source  selection  under 
consideration . 
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3-6b(6)  Evaluation  requirements. 

(b)  Critical  requirements  must  be  explained  in  the  RFP  with 
considerable  care.  These  requirements  may  relate  to  operational 
suitability,  performance ,  managenent,  schedule,  cost,  or  other  constraints 
that  could  finally  determine  whether  the  proposal  is  acceptable. 

3-9a  The  principal  purposes  of  proposal  evaluation.... 

(2)  Determine  from  among  the  proposals  received  (together  with  other 
factors  bearing  on  the  performance  potential  of  each  firm)  which  one 
provides  the  highest  probability  of  achieving  optimiiD  satisfaction  of  the 
Government's  objectives  including  technical,  schedule,  and  cost 
performance. 

(3)  Provide  a  sound  basis  for  the  SSA  to  make  an  informed  and 
objective  selection  by. ..(b)  Identifying  areas  of  uncertainty  and  risk  as 
well  as  those  in  which  there  is  substantial  assurance  of  a  successful 
outcome. 

c.  Proposal  evaluation  requires  a  mixture  of  factfinding,  reporting, 
and  the  application  of  professional  judgment  to  provide  a  rounded  and 
comprehensive  picture  of  the  adequacy  of  each  offer.  This  calls  for: 

(1)  Validation  of  the  representations,  estimates,  and  projections 
presented  in  each  proposal,  particularly  by  comparison  with  independent 
Government  estimates  of  performance,  schedule,  cost,  and  established 
requi rements . 

3-10a  The  component  tasks  of  the  evaluation  vary  in  number,  content, 
and  sequence  with  each  new  source  selection.... 

(3)  Verification  of  judgmental  aspects  of  a  proposal.... 

(d)  Another  form  of  verification  found  useful  is  to  cross-check  major 
features  in  the  technical  proposal  with  related  items  in  the  cost  proposal, 
management  proposal  ,  or  time  schedule. 

(5)  Scoring. 

(a)  Examining  each  proposal  in  detail  to  measure  the  predetermined 
elements,  factors,  and  items  against  the  established  criteria — and 
assigning  to  them  a  score,  numerical  or  otherwi se--constitutes  the  core  of 
the  evaluation  process.  The  effectiveness  of  prior  planning  and 
preparation  becomes  apparent  at  this  critical  stage  of  the  proposal 

eval  uation . 

(b)  Because  numerical  scores  or  other  types  of  grading  may  not  convey 
fully  the  individual  evaluator's  judgment  of  some  aspects  of  the  proposal, 
each  evaluator  must  supplement  his  rating  with  a  concise  narrative 
evaluation  which  includes  discussion  and  interpretation  of  the  limitations 
of  his  rating.  The  narrative  records  what  the  contractor  offered  and  how 
it  met  the  established  requirement,  and  it  summarizes  the  strong  and  weak 
points  of  what  the  contractor  has  proposed.  In  instances  where  the 
contractor  has  failed  to  meet  a  critical  requirement,  the  evaluator 
assesses  what  should  be  done  to  remedy  the  deficiency  and  what  the  impact 
of  the  deficiency  (corrected  or  uncorrected)  is  on  the  overall  proposal. 

The  numerical  and/or  narrative  evaluation  should  be  supplemented  with  a 
risk  assessment.  This  assessment  states  the  level  of  risk  inherent  in  the 
offeror  ultimately  achieving  the  capability  that  has  been  stated  in  the 
proposal . 
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(c)  All  errors,  omissions,  and  deficiencies  should  be  considered  by 
the  SSE8  evaluator  in  determining  the  initial  score  to  be  given  the 
criteria  or  factor  for  which  he  is  responsible,  unless  they  are  so  minor 
and  readily  correctable  that  his  committee  chairman  approves  discarding 
them  for  scoring  purposes. 

(7)  Conducting  the  negotiations. 

(a)  Negotiations  must  be  conducted  with  all  offerors  in  the 
competitive  range  unless  an  exception  or  qualification  of  this  requirement 
is  made  in  the  solicitation. 

(b)  Negotiations  are  generally  conducted  in  order  to  come  to  agreanent 
on  such  matters  as  cost  or  price  considerations ,  incentive  arranganents , 
special  provisions  to  be  incorporated  in  the  contractual  instrument  or 
scheduling.  Negotiations  are  also  frequently  required  to  resolve  any 
errors,  omissions,  or  deficiencies  that  were  identified  during  the  initial 
evaluation  prior  to  establishment  of  the  competitive  range. 

(d)  All  discussions  or  negotiations  with  offerors  within  the 
competitive  range  are  conducted  by  the  contracting  officer.  He  is  assisted 
by  such  technical  advisors  as  he  requires,  including,  in  some  instances, 
the  specialists  who  are  most  familiar  with  what  is  needed  to  remedy  errors 
and  correct  deficiencies. 

(8)  Best  and  final  offers. 

(a)  At  the  end  of  discussions/negotiations,  all  offerors  remaining  in 
the  competition  range  are  provided  one  final  opportunity  to  submit 
revisions  which  must  be  received  by  a  common  cutoff  date.  This  last 
opportunity  for  revision  is  known  as  the  best  and  final  offer. 

(9)  Final  evaluation  by  the  SSE8. 

(a)  When  the  revisions  to  proposals,  purporting  to  correct  errors, 
omissions,  and  deficiencies,  and  when  best  and  final  offers  are  returned  to 
the  SSE6,  those  portions  of  the  original  proposal  affected  require 
reevaluation  and  rescoring.  New  scores  are  then  computed  and  the  relative 
standing  of  the  competitors  determined  again. 

(10)  Summarization.  After  the  various  criteria  are  scored,  the  raw 
scores  must  be  summarized.  These  scores  are  in  diverse  forms;  ranging  from 
qualitative  assessments,  to  strengths  and  weaknesses,  to  go/no-go 
statements,  to  quantifications  of  adjectival  ratings,  to  actual 
quantitative  judgments  of  performance( i .e . ,  80  mph).  In  planning  the 
source  selection,  the  selection  plan  has  developed  provisions  for 
efficiently  summarizing  this  diverse  material  for  ranking  and  ultimate 
source-selection  decision.  In  turn,  the  SSAC  must  relate  the  evaluation 
criteria  with  weights,  priority,  or  trade-off  values,  or  by  describing  how 
judgmental  decisions  will  be  used  to  relate  the  raw  scores  to  a  summary 
level . 


3-14  The  source-selection  authority. 

c.  In  the  evaluation  and  selection  process,  the  selection  official  is 
not  bound  by  the  findings,  scoring,  and  recommendations  of  lower  level 
review  bodies  and  officials.  The  only  limits  on  the  discretion  of  the  SSA 
are  that  the  selection  must  have  a  rational  basis  in  terms  of  the 
evaluation  factors  in  the  RFP,  and  all  the  legal  and  procedural 
requirements  of  the  evaluation  process  must  be  met  (i.e.,  relative  order  of 
importance  of  factors  in  the  RFP,  competitive  range,  best  and  final  offers, 
etc.)....  The  final  decision  should  not  be  based  upon  numeric,  adjectival. 
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or  other  scores,  rather  it  should  be  based  upon  the  relative  strengths  and 
weaknesses  of  the  competing  proposals. 

3- 17  Debriefing  of  competitors. 

b.  Debriefings  are  not  intended  to  be  a  forim  for  debate,  and  the 
discussions  should  not  include  comparisons  with  other  proposals  but  should 
be  confined  to  the  specific  strengths  and  weaknesses  of  the  individual 
proposal  being  considered  versus  the  RF P  requirements.  An  explanation  of 
the  evaluation  method  is  appropriate  to  insure  the  unsuccessful  firms  that 
their  proposals  were  treated  fairly,  impartially,  and  objectively,  but  it 
would  be  inappropriate  to  disclose  specific  criteria  weights  or  the  rating 
scores  assigned  to  particular  companies. 

-  Chapter  4.  Special  Evaluation  Considerations 

4- 2  Considerations  in  selecting  criteria. 

a.  Proposal  evaluation  in  the  DOD  has  often  been  plagued  by  seemingly 
insol vable  recurrent  problems.  Probably  the  two  most  representati ve  of 
these  problems  are  the  selection  of  the  "wrong"  offeror  and  the  "dead 
heat."  That  is,  in  certain  situations  evaluators  are  driven  by 

ci rcimstances  to  recommend  award  to  an  offeror  who  does  not  have  the  best 
proposal  and  in  other  situations  evaluators  must  give  decisionmakers  two  or 
more  proposals  having  virtually  identical  evaluation  scores. 

b.  One  major  cause  for  these  recurrent  difficulties,  particularly  the 
first  one  mentioned  here,  is  simply  poor  planning.  The  PM  often  expresses 
a  general  evaluation  plan  (evaluation  factors  and  relative  weights  among 
them)  without  anticipating  any  problems.  However,  when  the  proposals  are 
received,  evaluators  may  find  that  as  a  result  of  their  plan,  they  have 
scored  highest  a  proposal  that  is  really  not  the  best,  or  that  they  cannot 
discriminate  among  proposals;  as  a  result  they  are  in  a  "box."  In  this 
situation,  the  PM  has  told  the  offerors  the  evaluation  rules  and  then  finds 
it  difficult  to  live  with  the  results  of  the  evaluation.  They  can  amend 
the  solicitation  to  change  the  evaluation  information  to  what  they  should 
have  originally  had  in  mind  to  discriminate  the  "best"  proposal,  but  this 
is  costly,  time-consuming,  and  may  draw  charges  of  favoritism  and  bias. 

They  can  resolicit  the  acquisition  and  endure  an  even  longer,  more  costly, 
and  perhaps  even  more  challengeable  evaluation.  They  can  go  ahead  with  the 
evaluation,  suggest  award  to  the  "wrong"  offeror  or  change  the  evaluation 
“rules"  d6  facto  and  award  the  "right"  one  and  in  either  case  suffer  a  true 
loss  of  objectivity  and  a  very  real  threat  of  a  protest.  The  alternatives 
in  any  case  are  not  good. 

c.  The  best  way  to  avoid  this  classic  dilemma  is  to  spend  more  time  on 
evaluation  planning.  The  PM  must  realize  the  first  attempt  at  an 
evaluation  plan  may  not  always  achieve  desired  results.  Evaluation  factors 
should  be  studied  and  chosen  with  great  care. 

4-3  Structuring  the  evaluation  matrix. 

b(l)  ...Three  major  areas  that  have  been  considered  more  or  less 
traditional  in  source  selection  are:  technical,  cost,  and  management; 
however,  there  is  nothing  sacred  in  such  a  division.  Other  areas  for 
considerations  are:  logistics,  operational  suitability,  test  and 
evaluation  (T&E),  etc.  The  evaluation  areas,  even  at  the  major  or  rollup 
level,  must  be  specifically  tailored  to  each  program,  and  the  needs  of  the 
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Army  which  that  program  is  seeking  to  satisfy. 

4-4  Relationship  of  the  criteria. 

a.  General.  Relativity  of  each  criterion  to  the  total  mission 
requirement  must  be  established  in  order  to  summarize  the  scores.  The 
decisionmaker  may  do  this  by  weighing  the  criteria,  by  priority  or 
trade-off  statements,  by  judgmental  decision  rules,  or  a  combination  of 
these.  Weighting  would  involve  an  assignment  of  relative  importance  among 
the  criteria  by  breaking  up  a  constant  sum  (typically  100  or  1,000  points). 
(See  DARC0M  Procurement  Instruction  3-501  for  weighting  informaton  to  be 
given  in  the  RFP.)  For  source  officials  who  do  not  want  specific  numerical 
relationships,  the  criteria  may  be  related  by  priority  statements  and  the 
scores  combined  by  judgment  using  these  priorities  rather  than  by  numeric 
formulas  using  weights.  For  example,  in  a  priority  statement  "cost''  may  be 
said  to  be  slightly  more  important  than  "management"  but  slightly  less  than 
"operational  suitability."  A  decision  rule  would  tell  how  to  deal  with  a 
factor  under  varying  conditions.  For  example,  a  decision  rule  might  be  "if 
management  is  rated  anything  less  than  satisfactory,  the  entire  proposal  is 
unacceptable,"  or  "if  the  proposed  price  is  30-percent  higher  than  the 
Government  estimate,  it  will  be  judged  as  being  potentially  unrealistic, 
the  cost  score  will  be  penalized,  and  the  technical  proposal  will  be 
reevaluated  to  see  if  there  is  some  misunderstanding  of  the  requirements." 
In  order,  then,  to  provide  a  base  for  the  evaluation,  a  precise  definition 
of  each  established  criterion  must  be  prepared  in  narrative  form  to 
indicate  what  it  is  and  how  it  is  to  be  used,  and  a  description  of  the 
alpha  or  numeric  standard  indicating  the  desired  performance  for  each  major 
element,  and  the  relationship  among  factors  and  subfactors.  Definitions 
should  be  either  included  in,  or  appended  to  the  selection  plan. 

b.  Weighting  guidance. 

(1)  In  the  process  of  source  selection,  a  weight  is  a  gauge  of  the 
relative  importance  of  all  the  areas,  elements,  and  factors  that  are  used 
in  evaluating  a  proposal.  Weights  applied  by  the  SSAC  to  the  scoring 
process  may  be  numerical ,  adjectival  ,  or  narrative.  One  must  be  alert  to 
the  fact  that  numerical  scoring  systems  do  not  insure  mathematical 
certitude.  Overreliance  upon  arithmetic  summari zation  could  lead  to  an 
incorrect  decision  based  on  the  tabulations  of  numerous  decisions  scored  by 
the  individual  evaluators.  Although  the  individual  scoring  (i.e., 
evaluation  of  the  proposal  at  element  and  factor  level)  may  be  quite 
accurate  and  acceptable  by  themselves,  it  is  inevitably  tied  to  a  system 
that  has  been  designed  in  gross  and  is,  therefore,  fallible. 

(2)  A  wide  range  of  criteria,  many  of  them  highly  specialized,  must  be 
identified,  reviewed,  and  evaluated  in  a  major  acquisition....  Each 
procurement  is  unique  in  itself,  and  the  various  evaluation  elements  and 
factors  are  of  greater  or  lesser  importance  in  each  case.... 

(3)  After  choosing  the  critical,  discriminating  criteria  evaluators 
(and/or  evaluation  decisionmakers)  must  decide  how  they  truly  feel  about 
the  relative  importance  among  the  areas  or  they  may  later  be  awarding  to 
the  "wrong"  offeror.  A  very  common  example  here  is  having  to  decide 
between  an  offeror  who  offers  more  performance  but  at  a  high  price  and  an 
offeror  who  offers  marginally  acceptable  performance  but  at  a  very 
attractive  price.  In  situations  of  low  technical  risk,  one  might  decide 
the  latter  offeror  could  handle  it  and  save  the  Government  some  money.  In 
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situations  of  high  technical  risk,  a  decisionmaker  may  decide  it  was  worth 
the  money  to  award  to  the  first  offeror  who  is  more  likely  to  succeed.  In 
either  situation,  the  decisionmaker  may  be  thwarted  because  improper 
weighting  has  "driven"  the  decision  to  the  wrong  offeror.  An  excellent 
high-performance  (but  high-cost)  proposal  may  be  favored  but  have  to  be 
passed  up  because  cost  was  weighted  too  high.  On  the  other  hand,  a 
technically  acceptable  (but  marginal)  proposal  with  an  attractive  price  may 
be  favored  in  other  ci rcunstances  but  lost  to  the  decisionmaker  oecause 
performance  was  weighted  too  high. 

(4)  Regardless  of  the  weighting  system  used,  it  is  essential  that  the 
distribution  of  weights  bear  a  discernable  relationship  to  valid 
identifiable  and  defensible  needs  of  the  acquisition.... 

(7)  The  relative  importance  of  weights  is  established  by  the  PM  in  the 
plan  in  a  form  for  use  in  the  proposal  solicitation.  Weights  must  not  be 
altered  once  the  proposals  have  been  received,  unless  an  SSAC  approved 
amendment  to  the  solicitation  is  issued. 

(a)  The  simplest,  and  the  most  commonly  used  method  is  weight 
assignment  by  percentage  distribution,  whereby  weights  may  be  distributed 
to  the  major  areas  or  elements  based  on  100  percent.  Within  each  of  the 
areas,  a  100-percent  distribution  may  be  employed  again  to  fix  the  relative 
importance  of  each  element.  If  desired,  this  technique  may  be  carried  down 
to  the  factor  levels.... 

(b)  Sound  contracting  policy  requires  that  prospective  offerors  be 
given  enough  information  regarding  evaluation  factors  and  their  relative 
weights  to  permit  them  to  prepare  their  proposals  properly.  Nonetheless, 
the  disclosure  of  exact  numerical  weights  could  create  an  inflexible 
situation  which  might  rigidify  the  contractor's  input  to  the  detriment  of 
its  overall  quality,  and  deny  to  the  SSA  the  measure  of  discretion  that  he 
needs  to  protect  the  Government's  interest.  Therefore,  the  RFP  should  make 
abundantly  clear  to  the  prospective  contractors  those  areas  requiring 
special  emphasis,  without  at  the  same  time,  disclosing  the  exact  numerical 
weights . 


4-5  Scoring . 

a.  The  actual  evaluation  of  proposals  is  accomplished  by  individual 
evaluators  judging  the  merits  of  each  criterion  identified  for  evaluation 
contained  in  the  contractor's  proposal  versus  the  Government  standards  to 
the  lowest  level  of  detail.  From  this  point  forward,  until  the  results  of 
the  detailed  evaluation  have  been  submitted  to  the  SSAC,  the  scoring 
process  is  simply  a  tabulation  of  factors,  elements,  and  major  area  scores 
into  a  composite  score  at  final  summarization  level.  As  an  additional 
point  of  reference,  a  contractor's  proposal  is  evaluated  by  comparing 
specific  parts  of  the  submitted  data  against  standards/ requi rements ;  not  by 
comparing  the  element  in  the  various  contractor's  submissions  with  each 
other.  The  numerical  ,  adjectival  ,  or  other  designator  developed  to  express 
how  well  the  standard  has  been  met  is  known  as  score,  or  rating. 

b.  The  most  frequent  used  rating  procedure  in  proposal  evaluation  is 
the  numeric  method.  Scores  are  normally  based  on  a  preestablished  scale 
from  0  to  10.  It  is  required,  however,  that  each  numerical  score  be 
accompanied  by  a  supporting  narrative  signed  by  the  evaluator  and 
discussing  strong  and  weak  points  which  were  considered  in  the  scoring. 
Reports  of  intermediate  groups  which  send  forward  composite  scores  should 
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be  accompanied  by  a  narrative  analysis  which  takes  into  account  both  the 
numerical  scores  and  the  individual  narratives. 

c.  Adjectival  scoring  is  often  employed  and  particularly  in  connection 
with  numerical  ratings.  Where  this  is  done,  it  is  customary  to  establish 
adjective  scoring  standards  in  advance,  for  the  purpose  of  having  the  same 
set  of  values  used  by  all  evaluators  in  arriving  at  their  independent 
judgments.  Atypical  numerical -adjectival  combined  rating  matrix  is  shown 
below. 

Score  Evaluation 

10  Excel lent--(innovative,  comprehensive  and  complete  in  all  details, 
meets  al 1  requirements  and  obectives  without  gold  plating). 

9  Very  Good-- ( suDstanti al  response  in  clearly  definable  detail,  meets 
all  critical  requirements). 

7  Average-- (general ly  meets  minimum  requirements). 

6  Poor--(lack  of  essential  information  to  substantiate  data  presented). 

5  Unsatisfactory--(lack  of  understanding  of  requirements  or  omissions 

in  major  areas)  . 

0  No  Data. 

d.  If  attendant  numerical  scoring  is  not  practical  ,  and  no  weighting 
factors  are  to  be  applied,  only  the  adjectival  ratings  above  can  be 

uti 1 i zed  . 

e.  Narrative  ratings  may  be  used  by  themselves,  or  combined  with 
numerical  scores  when  the  weighting  factor  is  to  be  applied.  When  a 
scoring  type  of  narrative  is  used  to  support  an  adjective  rating,  it  should 
be  based  on  the  same  criteria  as  the  adjective  rating  and  should  explain 
the  basis  for  either  high  or  low  scores. 

f.  Graphic  or  charting  techniques,  with  color  coding,  are  used  to 
reflect  relative  conclusions  and  are  particularly  effective  for  briefing 
purposes.  Necessarily  a  graphic  displaymust  be  supported  by  a  related 
narrative  presentation.  Graphic  evaluation  at  the  lowest  level  is  not 
suggested,  but  is  particularly  effective  at  the  summation  or  rollup  level. 

h.  Regardless  of  the  scoring  procedures  used,  once  established,  they 
must  be  impartially  applied  by  the  evaluators  to  each  proposal.  Any 
departure  from  the  established  norms  that  are  prompted  by  judgment  factors 
outside  the  procedure  is  proper,  only  insofar  as  the  same  treatment  is 
extended  on  an  impartial  basis  to  all  qualified  proposals  and  within  the 
limits  of  the  guidance  given  the  proposers  in  the  RFP. 

i.  Proposals  should  be  evaluated  as  submitted;  however,  errors, 
omissions,  and  deficiencies  are  handled  according  to  methods  described  in 
paragraph  3-10a(4). 

4-6  Special  factors. 

a.  In  every  selection  proceeding,  it  is  expected  that  there  will  be 
factors  that  require  appraisal,  which  either  do  not  fit  neatly  into  the 
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typical  pyramidal  evaluation  structure,  or  have  special  rules  for  their 
use.  Among  these  are  the  evaluation  of  life-cycle  costing,  human  factors 
engineering,  and  the  ratio  of  lowest  dollar  per  technical  quality  point. 

(1)  The  evaluation  of  proposer  life-cycle  cost. 

(d)  The  decision  to  use  life-cycle  cost  criteria  in  source  selection 
must  be  made  during  the  acquisition/contracting  planning  phase.  It  is 
undoubtedly  difficult  to  define,  quantify,  and  measure  many  of  the 
operating  and  support  costs  that  must  be  considered.  In  some  selections, 
it  may  be  impossible,  impractical,  or  too  costly  to  undertake  the  analyses 
needed  to  come  up  with  data  of  sufficient  reliability  to  be  used  as 
evaluation  criteria. 

(3)  Evaluation  of  lowest  dollar  per  technical  qual ity  point  ($/q.p). 
The  concept  of  selecting  the  best  technical  proposal  for  the  dollar  is 
supportable  by  the  dollar/technical  point  relationships.  This  factor  can 
be  determined  by  dividing  the  cost  of  the  proposal  by  the  total  unweighted 
raw  score  developed  for  "technical."  With  the  knowledge  contained  in  the 
RFP  that  this  relationship  will  be  given  consideration  and  will  force 
offerors  to  tradeoff  between  cost  and  technical  factors  in  order  to  prepare 
the  best  possible  proposal  at  a  fair  and  reasonable  price,  $/q.p. 
relationship  by  itself  is  not  justi f ication  for  selection;  but  is  only 
considered  an  additional  factor  to  offer  to  the  SSA  in  making  his  deciion. 

4-7  Summary. 

a.  ...Weighting  is  a  key  to  obtaining  the  desired  results  of  an 
evaluation  and  source  selection.  It  is,  however,  difficult  to  apply  when 
quantitative  scoring  is  not  used  and  the  evaluation  is  accomplished  on  only 
a  narrative  basis.  The  true  strengths  and  weaknesses  of  a  proposal  can  be 
clearly  revealed  by  a  narrative  evaluation  which  sets  forth  the  proposal's 
strengths  and  weaknesses.  In  the  absence  of  a  nuneric  score,  the 
application  of  a  numeric  weight  becomes  extremely  difficult.  Therefore,  a 
judgmental  conversion  of  narrative  score  to  numeric  must  be  made  before  the 
numeric  weight  can  be  applied.  Key  narrative  terminology  of  the  raw  score 
compared  with  numeral  interpretation  factors  will  provide  the  proper 
conversion. 

b.  Evaluators  must  avoid  surprises  by  anticipating  what  might  happen 
with  various  combinations  of  evaluation  factors,  weights,  and  types  of 
proposals.  In  other  words,  evaluation  planners  can  simulate  their 
procurement  by  writing  down  the  factors  and  weights  they  propose  to  use, 
making  up  simulated  proposals,  and  evaluating  these  proposals  against  the 
tentative  factors.  Then  it  can  be  seen  what  kind  of  proposal  is  "favored," 
whether  a  winner  can  be  discriminated,  and  whether  the  SSAC  could  adjust 
the  factors  and  weights  to  achieve  the  kind  of  results  they  really  want. 


DARCOM-Ci rcul ar  715-8-80 
Procurement  Instruction 
9  October  1980 

Excerpts  related  to  cost  evaluation: 

-  Appendix  A,  Section  M.  Evaluation  Factors  for  Award. 

(d)  ...The  following  is  an  example  of  a  provision  relating  to 
eval uation  of  cost : 

EVALUATION  OF  COST.  This  element  includes  the  evalation  of  the  target 
cost  and  target  fee  proposed  by  the  offeror  for  performing  all  the 
requirements  of  the  contemplated  contract  as  set  forth  in  this  RFP.  The 
evaluation  will  include  an  analysis  of  all  costs  proposed  and  submitted  on 
DD  Form  663  (Contract  Pricing  Proposal),  together  with  all  supporting  cost 
information  and  data.  The  offeror's  proposed  costs  will  be  eval uated  by 
comparison  with  the  Government  cost  estimate  and  by  appropriate 
consideration  of  information  from  the  Defense  Contract  Audit  Agency, 
Government  Technical  Personnel,  and  other  sources.  The  offeror's  price 
will  then  be  evaluated  in  accordance  with  its  comparative  advantage  to  the 
Government . 
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AF  Regulation  70-15 

Procurement  Source  Selection  Policy  and  Procedures 
16  April  1976 

Excerpts  related  to  cost  evaluation: 

-  Table  of  Contents 

This  regulation  implements  DOD  Directive  4105.62,  6  Jan  76,  and  is 
consistent  with  current  systems  acquisition/program  management  policies  in 
AFR  800-2. 

-  Chapter  1.  General  Information  and  Policy 

1-1.  Applicability  and  Scope. 

a.  The  source  selection  policies  and  procedures  set  forth  in  this 
regulation  apply  to  the  following  competitive  procurements  : 

(1)  Each  new  development  program  estimated  to  require  $100  million 
or  more  RDT&E  funds  or  projected  to  require  $500  million  or  more  production 
funds  (including  support).  These  policies  and  procedures  will  be  used  to 
select  the  source(s)  for  the  Advanced  Development,  Validation  and 
Full-Scale  Development  contracts. 

b.  Policies  and  procedures  contained  in  this  regulation  are 
sufficiently  flexible  to  accommodate  a  wide  range  of  requirements.  They, 
therefore,  may  be  used  as  a  guide  to  formally  evaluate  competitive 
proposal s  and  to  select  sources  for  other  programs/projects  below  the 
dollar  thresholds  prescribed  above.  In  this  context,  these  procedures 
should  be  tailored  to  individual  prog ram/ project  requirements  and 
selectively  applied  to  avoid  excessive  costs. 

1-2.  Objectives.  The  prime  objectives  of  the  process  of  proposal 
solicitation,  evaluation  and  source  selection  are  to  select  the  source 
whose  proposal  has  the  highest  degree  of  realism  and  creditability  and 
whose  performance  is  expected  to  best  meet  Government  objectives  at  an 
affordable  cost;  assure  impartial,  equitable,  and  comprehensive  evaluation 
of  competitor's  proposals  and  related  capabilities;  and  effect  maximum 
efficiency  and  mimimum  complexity  of  proposal  solicitation,  evaluation  and 
the  selection  decision. 

1-3.  Explanation  of  Terms. 

j.  Evaluation  Criteria.  The  standards  used  in  the  Source  Selection 
process  to  measure  proposal  acceptability  in  regard  to  the  Government 
requirements  stated  in  the  Request  for  Proposal. 

l.  Independent  Cost  Estimate  (ICE).  The  Government's  cost  estimate 
used  for  program  approvals,  for  establishing  budget  estimates,  and  as  a 
baseline  for  evaluating  offerors'  proposals.  The  ICE  should  be  an 
independent,  objective,  unbiased  estimate,  which  stands  on  its  own  as  a 
"yardstick"  for  evaluating  offerors'  proposals.  The  full  spectrum  of 
quantitative  tools  and  techniques  should  be  brought  to  bear  in  developing 
the  ICE. 

m.  Life  Cycle  Cost.  The  total  cost  of  an  item  or  system  over  its  full 
life.  It  includes  the  cost  of  development,  acquisition,  ownership 
(operation,  maintenance,  etc.)  and  where  applicable,  disposal  (see  AFR 
800-11). 
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n.  Most  Probable  Cost  (MPC).  The  Government  estimate  of  the  total  cost 
most  likely  to  be  incurred  by  each  of  the  competing  offerors  if  a  contract 
is  awarded. 

ad.  Source  Selection  Process.  The  formalized  process  employed  in 
competitive,  negotiated  procurements  (other  than  for  proposed  awards  to  be 
based  primarily  on  price  competition)  for  programs/projects  as  defined  in 
paragraph  1-1.  The  source  selection  is  designed  to  insure  an  impartial, 
equitable,  and  economic,  evaluation,  and  comparative  analysis  of  competing 
offerors'  proposals  and  their  capabilities.  This  process  provides  an 
objective  basis  for  the  SSA  to  select  the  source(s)  which  best  satisfies 
the  Government's  requi rements  . 

1-4.  Policies. 

d.  The  source  selection  process  shall  focus  adequate  attention  on  the 
program  risk  and  uncertainties  during  solicitation,  proposal  evaluation, 
and  selection  phases. 

(1)  Offerors  should  not  be  penalized  for  the  identification  of 
risk  associated  with  their  proposals.  Proposals  should  be  credited  when 
realistic  approaches  for  risk  resolution  are  provided. 

(2)  The  procuring  activity  shall  prepare  an  independent  risk 
assessment  before  receipt  of  proposals,  to  facilitate  risk  analysis 
eval  uati  on . 

e.  The  solicitation  document  shall  identify  areas  of  potential  high 
technical  and  manufacturing  risks  when  there  is  reason  to  believe  that  the 
risks  are  not  generally  known  to  the  offerors.  Offerors  will  be  required 
to  identify  the  risk  associated  with  their  proposals  in  the  areas  of  cost, 
schedule,  or  performance,  together  with  realistic  approaches  for  resolving 
or  avoiding  the  identified  risks  (including  backup  development  efforts). 

f.  The  solicitation  document  will  encourage  identification  of 
potential  trade-offs  across  the  spectrum  of  design,  development,  production 
and  operational  and  support.  All  new  major  defense  systems  will  include 
appropriate  design-to-cost  goals.  These  goals  must  be  clearly  defined  in 
the  solicitation  document  and  must  be  consistent  with  the  total  cost 
approach  so  as  to  encourage  trade-off  between  acquisition  cost  and  life 
cycle  cost  (reliability,  maintainability  and  other  logistics  cost). 

i.  Evaluation  criteria.... 

(1)  The  specific  evaluation  criteria  must  be  included  in  the 
solicitation  document  and  enumerated  in  terms  of  relative  order  of 
importance  of  those  significant  factors  which  will  form  the  general  basis 
for  proposal  evaluation  and  selection/contract  award. 

(3)  The  solicitation  shall  advise  each  potential  offeror  that  his 
proposal  is  presumed  to  represent  his  best  efforts  to  respond  to  the 
solicitation,  and  any  apparent  inconsistency  between  promised  performance 
and  cost  or  price  should  be  explained.  Further,  any  significant 
inconsistency,  if  unexplained,  raises  a  fundamental  issue  of  the  offeror's 
understanding  of  the  work  required  and/or  of  his  ability  to  perform  the 
contract  and  may  be  grounds  for  rejection  of  the  proposal  ,  or  a  basis  for 
determination  of  nonresponsibility.  The  solicitation  shall  also  advise 
that  the  burden  of  proof  of  cost  credibility  rests  with  the  offeror. 

j.  The  rating  system  or  techniques  to  be  used  in  evaluating  and 
analyzing  proposals,  shall  be  set  forth  in  the  Source  Selection  Plan  for 
approval  by  the  SSA.  The  rating  system  shall  be  structured  to  enable  the 
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SSA  to  identifiy  the  significant  differences,  strengths,  weaknesses,  and 
risks  associated  with  each  proposal  and  subsequent  definitized  contract. 

n.  The  rating  system  may  be  entirely  narrative,  or  may  employ 
numerical  scoring  and  weights  or  a  descriptive  color  code  in  conjunction 
with  narrative  assessments  (ref  para  3-10).  The  important  task  in  either 
rating  system  is  the  integrated  assessment  of  all  aspects  of  the 
evaluation,  analysis,  and  negotiation  process. 

q.  Continued  effort  must  be  exerted  to  achieve  greater  cost  realism  in 
proposals  for  major  development  programs.  In  order  to  achieve  this 
obj ective: 

(1)  The  preparation  of  Independent  Cost  Estimates  (ICE),  along 
with  the  use  of  most  probable  cost  projections  are  to  be  used  during  source 
selection  to  assess  cost  realism  during  the  evaluation  of  offerors' 
proposal  s. 

(a)  As  a  matter  of  policy,  however,  release  of  Independent 
Cost  Estimates  (ICE)  is  not  advisable,  as  the  recipient  may  pass  the 
information  on  to  companies  that  will,  in  effect,  be  offering  against  the 
ICE. 

(b)  The  extent  to  which  such  a  transmission  of  data  will 
encourage  unsubstantiated  reductions  in  pricing  proposals  on  the  part  of 
those  contractors  is  difficult  to  predict,  but  the  incentive  is  clear. 

(2)  Auction  techniques  (indicating  to  an  offeror  a  price  which 
must  be  met  to  obtain  further  consideration,  or  informing  him  that  his 
price  is  not  low  in  relation  to  that  of  another  offeror)  are  strictly 
prohibited.  On  the  other  hand,  this  does  not  prohibit  pointing  out  price 
or  cost  elements  that  do  not  appear  to  be  justified,  or  encouraging 
offerors  to  put  forward  their  most  favorable  price  proposals;  but  in  so 
doing  the  price  elements  of  any  other  offeror  must  not  be  discussed, 
disclosed,  or  compared. 

(3)  When  a  cost/price  proposal  appears  to  be  unrealistically  high 
or  low,  a  special  assessment  shall  be  included  in  the  SSAC  analysis  report 
and  an  oral  briefing  prepared  addressing  the  rationale  for  the  apparent 
unrealistic  cost/price.  As  prescribed  in  ASPR  4-106. 4(c),  a  proposal  may 
be  penalized  in  this  special  assessment  to  the  degree  that  the  proposed 
cost/price  is  unrealistically  low. 

1-6.  Organization. 

d.  (1)  Due  to  the  competitive  environment  in  which  discussions  are 
conducted  during  the  proposal  evaluation  and  source  selection  period, 
offerors  may  tend  to  accept  deficiency  notifications  as  being  a  mandatory 
requirement  to  change  their  proposal. 

(a)  Each  offeror  must  be  informed  that  deficiencies  are  presented 
as  advisory  guidance  only  and  it  is  at  his  election  whether  or  not  to  make 
a  change  in  the  proposal.  He  does  not  have  to  agree  with  the  Government. 

(b)  In  pointing  out  deficiencies,  the  Contract  Definiti zation 
Team  will  not  suggest  ways  of  improving  the  proposal,  since  offerors  shall 
not  be  given  the  benefit  of  Government  assistance  which  other  offerors  in 
the  competitive  range  do  not  need  or  otherwise  receive. 

e.  In  discussing  costs/prices,  the  Contract  Definitization  Team  should 
recognize  that  under  the  constraint  of  competition,  over-zealous  cost/price 
discussions  may  cause  offerors  to  lower  their  costs/prices  to  a  point 
where,  if  accepted,  could  place  the  program  in  financial  jeopardy  even 
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before  work  is  begun.  Government  negotiators  must  not  lose  sight  of  their 
own  independent  cost  estimates,  most  probable  cost  estimates,  and  other 
techniques  used  during  these  discussions  which  help  to  determine  cost 
realism.  While  it  is  imperative  that  unrealistically  low  or  high  cost 
elements  be  properly  investigated  and  discussed,  there  is  no  intent  that 
the  discussion  should  lower  a  price  to  a  point  that  it  is  unrealistic 
through  the  pressures  of  competition. 

1- 8.  Determination  of  Competitive  Range,  Conducting  Written  or  Oral 
Discussions,  and  Deficiency  Resolution. 

e.  All  offerors  determined  to  be  in  the  competitive  range  and  selected 
to  participate  in  oral  or  written  discussions  must  be  advised  of  any 
deficiencies  in  their  proposals,  and  offered  a  reasonable  opportunity  to 
correct  or  resolve  the  deficiencies.  Offerors  must  submit  such  price/cost, 
technical  ,  or  other  proposal  revisions  as  may  result  from  the  discussions. 

(1)  For  the  purpose  of  conducting  discussions  with  offerors 
determined  to  be  within  the  competitive  range,  a  deficiency  is  defined  as 
any  part  of  an  offeror's  proposal  which  would  not  satisfy  the  Government's 
minimum  requirements  as  stated  in  the  solicitation  document. 

f.  (4)  When  Contracting  Officers  call  for  the  "best  and  final"  offer, 
offerors  should  be  cautioned  against  buying-in  and  submitting  unsupported 
changes  in  their  former  offers.  This  can  be  accomplished  by  referencing 

AS  PR  1-311  and  the  appropriate  provision  of  the  solicitation  document  which 
establishes  the  relative  importance  of  cost  in  the  evaluation  process. 

-  Chapter  2.  The  Source  Selection  Process 

2- 2.  Source  Selection  Plan. 

b.  (6)  Evaluation  of  Costs: 

(a)  Identify  those  items  of  the  system  or  subsystem  (Military 
Procurement  Code,  Work  Breakdown  Structure)  for  which  costs  will  be 
evaluated.  Such  items  as  overhead  that  aro  considered  to  have  sufficient 
cost  impact  to  warrant  special  consideration  will  be  separately  identified. 
Provisions  for  economic  fluctuations  and  the  rationale  for  estimating 
anticipated  price  level  changes  will  be  included,  so  that  cost  estimates 
>.an  be  evaluated  in  both  base  year  and  then  year  price  levels.  Items  that 
are  cost  related  and  have  an  economic  impact  on  the  program  but  which  are 
not  quantifiable  will  also  be  identified.  Plans  for  the  use  of  in-house 
configurations  or  designs  in  developing  the  ICE  should  also  be  presented. 

(b)  Outline  the  steps  to  be  followed  to  formulate  the 
Government's  best  estimate  of  the  total  cost  to  the  Government.  This  will 
include  the  methodology,  scope,  and  data  to  be  used  in  preparing  the 
independent  cost  estimate  and  how  it  will  be  used  in  evaluating  the 
offerors'  proposals. 

2-4.  Solicitations. 

a.  (6)  Unrealistically  low  estimates,  initially  or  subsequently,  may 
be  grounds  for  eliminating  a  proposal  from  competition  either  on  the 
grounds  that  the  offeror  does  not  understand  the  requirement  or  that  the 
offeror  has  made  an  improvident  proposal.  The  burden  of  proof  of  cost 
credibility  rests  with  the  offerors.  If  it  is  determined  that  an  offeror's 
estimates  are  unexpl ainably  low,  that  offeror  may  be  eliminated  from  the 
competition  (subject  to  the  requirements  of  ASPR  3-805).  Notice  to  this 
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effect  should  be  set  forth  in  the  sol icitation. 

d.  Offerors  will  be  required  to  identify  the  risk  associated  with 
their  proposals  in  the  areas  of  technical,  cost,  schedule,  or  performance, 
together  with  realistic  approaches  for  resolving  or  avoiding  the  identified 
risks  (including  backup  development  efforts).  Risk  analysis  is  a  part  of 
the  evaluation  process,  and  risk  assessments  for  each  proposal  must  be 
included  in  all  reports  to  the  SSAC  and  SSA.... 

-  Chapter  3.  Evaluation  and  Comparative  Analysis 

3-1.  General . 

b.  ...The  techniques  of  evaluation  and  analysis  will  vary  with  the 
acquisition  approach  employed  and  the  specific  requirements  of  the  program, 
which  may  be  to  procure  an  end  item,  a  design,  a  technical  approach,  or  a 
design  team  effort. 

c.  Depending  on  the  objectives  of  the  procurement,  the  source 
selection  process  must  be  sufficiently  flexible  to  accommodate  these 
requirements.  This  will  require  that  the  approach  to  source  selection  and 
the  development  of  the  evaluation  criteria  be  tailored  to  the  specific 
needs  of  whatever  is  being  procured  under  the  contract. 

3-2.  Evaluation  Criteria. 

a.  (3)  The  evaluation  criteria  contained  in  the  solicitation  must  set 
forth  the  relative  importance  among  technical,  costs  (development  versus 
production  versus  operational  and  support  costs),  schedule,  management  and 
other  factors  as  set  forth  in  the  Source  Selection  Plan.  In  the  event  that 
requirements  or  conditions  significantly  change  so  as  to  negate  or  modify 
the  evaluation  criteria  originally  established  in  the  solicitation,  the  SSA 
shall  ensure  that  each  potential  offeror  is  informed  by  the  PCO  in  the  form 
of  an  RFP  amendment  of  the  adjusted  criteria  and  basis  for  award,  and  that 
sufficient  time  is  provided  for  modification  to  the  initial  proposal. 

b.  (1)  The  policy  emphasizes  that  the  evaluation  criteria  should  be 
tailored  to  the  characteristics  of  a  particular  program/project  and  should 
include  only  those  significant  aspects  which  will  have  an  impact  on  the 
ultimate  selection  decision. 

(2)  ...The  approach  should  be  one  of  selectivity  in  developing 
realistic,  pertinent,  and  significant  criteria  which,  when  applied,  will 
highlight  the  strengths,  weaknesses,  and  risks  of  each  proposal. 

3-3.  Specific  Evaluation  Criteria  (Areas  and  Items). 

c.  (2)  Through  experience  the  most  orderly  breakout  of  areas  is  as 
follows:  technical,  operations,  logistics,  management/production,  and 
costs.  This  exact  structuring  is  not  mandatory  in  all  cases  and  should  be 
adjusted  to  accommodate  the  specific  requirement. 

g.  In  each  source  selection  there  are  certain  characteristics  which 
must  be  evaluated.  These  are: 

(1)  degree  of  technical  risk  contained  in  the  offeror's  proposal; 

(2)  the  confidence  that  the  Government  can  place  in  the  offeror's 
cost  proposal ;  and 

(3)  the  financial  capability  of  the  offeror  in  relation  to  the 
financial  risk  associated  with  the  proposed  task  and  type  of  contract 
proposed. 

(4)  The  evaluation  of  these  criteria  requires  subjective  judgment 


C-38 


as  they  may  not  lend  themselves  to  numerical  scoring.  However,  they  must 
necessarily  be  fully  addressed  in  narrative  form,  both  by  the  SSEB  and  the 
SSAC.  Government  confidence  in  offeror's  cost  proposal  and  his  financial 
capability  best  lend  themselves  to  evaluation  under  the  cost  area  which  is 
not  to  be  scored  .lumerical  ly.  They  must,  accordingly,  be  addressed  in 
narrative  form. 

3-6.  Evaluation. 

d.  The  preparation  of  the  evaluation  analysis  in  narrative  form  is  the 
most  important  aspect  of  tne  evaluation  process. 

(1)  In  preparing  the  narrative  which  communicates  the  evaluator's 
findings  to  the  SSAC,  he  must  understand  that  his  narrative  will  prove  the 
most  useful  ,  and  usually  the  only  means  available  to  the  SSAC  to  inform  the 
SSA  what  a  company  offers  and  how  well  it  met  the  established  standards. 

(2)  It  is  not  sufficient  that  the  narrative  states  that  something 
is  good  or  inferior.  The  evaluator  must  indicate  in  the  narrative  what  was 
offered;  how  it  met  the  standard,  or  how  it  failed  to  meet  the  minimum 
requirements;  what,  in  the  evaluator's  opinion,  must  be  done  to  remedy  the 
deficiency;  and  what  impact  (including  technical  and  cost  risk)  the 
deficiency  correction  will  have  on  the  overall  acceptabil  ity  of  the 
offeror' s  proposal . 

(3)  Clarity  and  brevity  are  the  keys  to  successfully  prepared 
narrati ves . 


3-7.  Assessment  of  Risk. 

a.  Identification  and  assessment  of  risk  are  essential  parts  of  the 
evaluation  process. 

(1)  Certain  risks  may  be  inherent  in  a  program  by  virtue  of  the 
program  objectives  relative  to  the  state  of  the  art.  Risks  may  occur  as  a 
.esult  of  a  particular  technical  approach,  manufacturing  plan,  the 
selection  of  certain  materials,  processes,  equipment,  etc.,  and  as  a  result 
of  the  costs  and  schedules  and  economic  impacts  associated  with  those 
approaches . 

(2)  Additional  risks  become  apparent  when  a  given  design  approach 
results  in  marginal  operational  performance. 

b.  Certain  risks  may  be  known  to  exist  in  major  programs  at  the  time 
the  solicitation  is  transmitted  to  industry.  In  this  case,  the  SSEB  and 
SSAC  must  not  penalize  proposals  merely  because  of  the  existence  of  these 
particular  risks. 

(1)  Standards  prepared  for  the  items  and  factors  where  known  risks 
exist  must  be  such  that  they  can  be  used  to  measure  acceptabil ity  of  the 
proposed  solution,  rather  than  place  undue  emphasis  on  the  existence  of  the 
risk. 

(2)  Measurement  of  acceptability  must  consider  both  the  approach 
proposed,  the  alternatives  available,  and  any  backup  solutions  described  in 
the  proposal . 

c.  In  the  judgment  and  analysis  of  risks  the  evaluator  must  consider 
the  probability  of  success  and,  in  event  of  failure,  what  alternatives  are 
available  to  overcome  the  risks.  Further,  the  assessment  must  determine 
when  success  or  failure  will  be  apparent  and  the  impact  the  correction  will 
have  at  that  point  in  time.  In  this  case,  examination  of  the  alternatives 
available  and  the  ability  of  the  alternatives  to  meet  the  minimum 
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requirement  must  be  analyzed  and  reported. 

d.  In  order  that  proper  consideration  may  be  given  to  risk,  the 
evaluation  criteria  for  all  programs  must  include  overall  risk  assessment 
for  each  proposal  as  a  rated  item  or  area  in  the  evaluation  process.  Areas 
of  potential  high  risk  must  be  identified  in  the  solicitation  when  there  is 
reason  to  believe  that  the  risks  are  not  generally  known  to  the  contractor. 

e.  The  offerors  must  be  required  to  submit  a  risk  analysis  as  a  part 
of  their  proposal  which  also  identifies  risk  areas  and  which  furnishes  an 
insight  to  the  evaluator  as  to  how  the  offeror  intends  to  resolve  these 
risks  and  the  alternatives  to  overcoming  the  high  risk  approaches. 

(1)  In  order  to  aid  the  evaluator  in  performing  the  risk  analysis, 
the  procuring  activity  should  prepare  an  independent  risk  assessment  prior 
to  receipt  of  proposals. 

(2)  The  SSAC  must  include  in  its  analysis  report  and  presentations 
to  the  SSA,  an  assessment  of  the  cost,  technical,  management,  production, 
schedule  and  economic  risks  associated  with  each  proposal.  This  assessment 
will  be  based  on  an  evaluation  of  the  offeror's  risk  analysis  and  an 
evaluation  of  the  specific  design  proposed. 

f.  The  technical  risk  may  impact  operational  capabilities,  program 
schedules,  and  costs/prices.  It  is  imperative  that  these  risk  impacts  on 
costs/prices  be  properly  considered  in  the  determination  of  most  probable 
cost . 

(1)  It  is  the  responsibility  of  the  technical  evaluation  teams  to 
insure  that  the  cost  team  is  informed  of  the  risk  area,  and  further,  that 
the  cost  team  is  aided  in  arriving  at  a  judgment  of  cost  impact  which  may 
result  therefrom. 

(2)  The  impact  of  technical  risk  on  costs/prices  must  be  included 
in  the  SSAC  analysis  report  to  the  SSA. 

3-8.  Evaluation  of  Costs/Prices. 

a.  The  total  elements  evaluated  in  conducting  an  analysis  of 
cost/price  varies  with  each  prog  ram/ project.  It  is  essential  that  the  cost 
information  included  in  the  SSAC  analysis  report  in  narrative  format  and 
presentation  to  the  SSA  be  in  sufficient  depth  to  support  the  selection 
decision.  Costs  will  not  be  numerically  scored. 

b.  Cost/price  evaluations  of  proposals  generally  require  the 
f  o  1  lowi  ng  : 

(1)  Assurance  of  comparabil ity.  Consideration  must  be  given  to 
variations  in  amount  of  Government-furnished  property  (GFP)  or  materials, 
use  of  Government-owned  facilities  and  tooling,  and  all  other 
inconsistencies  before  the  offerors'  proposals  can  be  equitably  evaluated. 

(2)  Verification  of  rates.  This  requires  a  determination  that 
labor  and  overhead  rates,  and  any  special  pricing  factors  are  reasonable, 
allocable  and  consistent  with  satisfactory  accounting  and  estimating 
systems.  (Defense  Contract  Audit  Agency  (DCAA)  and  Contract  Administration 
Office  (CAO)  personnel  should  be  called  upon  to  assist  in  making  this 
determination.) 

(3)  Determination  of  cost/price  realism.  Cost/price  realism  will 
be  judged  by  the  use  of  cost/price  analysis.  Wide  disparities  must  be 
analyzed,  documented  and  reported  in  the  analyis.  Further  measurement  of 
cost/price  realism  will  be  made  by  comparing  most  probable  costs  with 
proposed  cost/prices  after  considering  the  risk  associated  with  the  design 


C-4D 


approach,  and  disposition  of  deficiencies.  Full  use  should  be  made  of  the 
Field  Price  Analysis  Report  prepared  by  the  Contract  Administration  Office 
under  ASPR  Section  III,  Part  8. 

(4)  Determination  of  cost  impact  as  a  result  of  deficiency 
disposition.  After  offeror  responds  to  Government-identified  deficiencies 
(that)  have  been  analyzed,  all  cost/price  impacts  resulting  therefrom 
should  again  be  considered  by  the  cost  team  in  their  determination  of 
cost/price  real  ism. 

(5)  Determination  of  cost/price  risk.  Based  on  the  data  developed 
above  and  the  technical  risk  assessments  for  each  proposal  ,  evaluate  the 
cost/price  risk  inherent  in  each  proposal.  To  the  extent  practicable,  the 
potential  impact  of  the  identified  risk  will  be  reflected  in  the  SSAC 
Comparative  Analysis  Report. 

(6)  Development  of  cost  track.  The  cost/price  team  should 
maintain  a  cost  track  to  facilitate  the  understanding  of  various  changes 
leading  to  the  final  cost/price  review  by  the  SSA.  The  cost  track  will  be 
used  for  establishing  a  cost  baseline  for  evaluating  post  contract  award 
changes . 

(7)  Preparation  of  a  special  assessment  for  any  cost/price  which 
appears  to  be  unrealistic.  The  assessment  should  address  the  reasons  why 
the  SSEB  believes  the  estimate  to  be  unrealistic. 

3-10.  Use  of  Rating  or  Scoring  Techniques  (as  Applicable). 

a.  Having  assessed  the  offerors'  data  and  recorded  the  assessment  of 
the  factor  or  subfactor  by  means  of  a  narrative  as  evaluated  against  the 
standard,  it  may  be  useful  as  an  aid  for  evaluation  to  develop  a 
quantitative  rating  which  indicates  how  well  the  offerors'  proposals  as 
initially  submitted  met  the  minimitn  requirements  expressed  in  the 
solicitation.  This  should  be  done  prior  to  any  action  relative  to 
correction  of  deficiencies. 

b.  It  is  not  absolutely  essential  to  score  proposals  if  the  items  and 
factors  being  evaluated  do  not  lend  themselves  to  numerical  scoring. 

(1)  For  example,  costs,  financial  capability,  and  other  aspects 
of  a  contractor's  management/ production  capability  may  not  lend  themselves 
to  scoring. 

(2)  Costs  in  fact  will  not  be  numerically  scored,  and  evaluation 
in  narrative  form  in  the  management  or  production  areas  may  be  adequate  to 
portray  to  the  SSAC/ SSA  the  significant  strengths  and  weaknesses  of  each 
response  in  these  areas. 

h.  Whether  numerical  scores  and  weights  or  color  codes  are  employed, 
it  must  be  recognized  that  they  are  communications  techniques  and  are  but 
one  of  the  factors  to  be  considered  by  the  SSA  in  reaching  his  final 
selection  decision.  Their  relative  importance  must  be  kept  in  proper 
balance  with  all  other  salient  factors  and  must  be  fully  supported  with 
narrative  rationale. 

i.  When  numerical  or  color  code  scoring  is  employed,  proposals  must 
only  be  scored  as  originally  submitted,  to  encourage  the  best  initial 
submission  from  the  offerors.  Proposals  must  not  be  rescored  as  result  of 
written  or  oral  discussions.  However,  any  changes  from  the  original 
proposal  should  be  identified  in  the  discussion  summary  and  analyzed  for 
the  SSA  in  the  SSAC  Analysis  Report  and  oral  presentation. 

k.  In  order  to  maintain  the  integrity  of  the  source  selection  and 
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ensure  that  the  best  proposals  are  submitted  initially,  numerical  scores 
(if  used)  will  be  based  on  the  original  submission. 


-  Appendix.  SSAC  Proposal  Analysis  Report. 

IV.  Cost  to  the  Government.  The  reasonableness,  realism,  and 
completeness  of  each  contractor's  proposal  should  be  fully  explained.  This 
section  normally  includes  data  pertaining  to  the  cost/price  analysis, 
independent  cost  estimates,  total  costs  to  the  Government,  most  probable 
cost,  impact  of  technical  uncertainty  on  cost/price,  etc.  This  section 
will  also  include  an  explanation  of  the  relative  importance  of  costs  in 
relation  to  other  aspects  of  the  program  and  any  special  assessments  made 
in  relation  to  unrealistic  cost/price  proposals. 

VIII.  Risk  Analysis.  Discuss  the  overall  impact  of  all  significant 
risks  and/or  "soft  spots"  associated  with  each  proposal.  These  will 
incl ude : 

a.  Technial  risks  inherent  in  the  offeror's  proposal  including 
technical  capability  (whether  demonstrated  or  not); 

b.  Confidence  that  can  be  placed  in  the  cost/price  estimate  provided 
by  each  offeror  considering  technical  risk; 

c.  Schedule  risk  as  assessed  against  the  technical  approach  and  the 
prevailing  economic  environment  (material  shortages); 

d.  The  financial  risk  to  each  offeror  in  relation  to  the  type  of 
contract  and  task  involved;  and 

e.  Production  risks  relating  to  make-or-buy  decisions,  anticipated  new 
manufacturing  technologies,  availability  of  production  facilities,  and 
overall  production  competence.  Any  design  trade-offs  made  by  the  offerors 
and  their  impact  on  costs  should  also  be  discussed. 

-  Appendix.  SSAC  Presentation  to  Commanders,  AF  Council,  Chief  of  Staff, 
Assistant  Secretaries,  and  the  Secretary. 

d.  Cost  to  the  Government.  This  section  of  the  presentation  shows  the 
cost/price  as  originally  proposed,  as  adjusted  for  comparability;  compares 
the  ICE  with  adjusted  cost/price;  indicates  most  probable  cost  based  on  Air 
Force  estimates  and  hignlights  major  cost  risks.  Where  major  differences 
are  apparent  between  offeror  proposed  and  Air  Force  estimated  cost, 
probable  financial  impact  on  the  offeror  is  indicated. 

f.  Risk  Analysis.  This  part  shows  an  overall  analysis,  and  the 
resultant  impact  of  risks  of  each  proposal.  This  should  include  the 
offeror's  proposed  solution  to  known  risks,  the  impact  of  risks  resulting 
from  peculiar  approaches  chosen  by  the  offeror,  the  impact  these  risks  have 
on  the  program  and  associated  costs/prices,  and  the  financial  risks  to  be 
taken  by  the  offeror  in  relation  to  the  type  of  contract  as  well  as  any 
overal 1  risks  to  the  program  accrued  through  these  risks. 
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AFSC  Regulation  80-15 

R&D  Source  Selection  Policy  and  Guidance 

31  December  1974 

Excerpts  related  to  cost  evaluation: 

-  Chapter  1.  General  Procedures  and  Guidance 

1-1.  Purpose  and  Objectives. 

b. (2)  Ensure  that  the  source  selection  process  is  flexible  enough  to 
accommodate  the  objectives  of  both  the  R&D  program  and  an  appropriate 
contractual  relationship.  Ensure  the  acquisition  approaches  and  procedures 
allow  seeking  the  best  scientific  and  technical  sources  which  also  results 
in  an  award  with  the  balanced  mix  of  performance,  cost,  and  schedule. 

c.  The  regulation  addresses  R&D  procurements  consisting  of  studies, 
projects  exploring  solutions  for  technical/scientific  concepts,  and 
programs  requiring  design  and  development  of  hardware  items  for  test  and 
experimentation.  These  procurements  must  be  evaluated  under  the  guidance 
of  ASPR  4-106.4,  which  provides  major  emphasis  to  technical  competence  but 
requires  consideration  of  all  other  pertinent  factors  including  management 
capabilities,  cost  controls,  and  past  performance  in  determining  the  offer 
selected  for  award. 

1- 2.  Applicability  and  Scope. 

a.  This  regulation  is  not  intended  for  use  on  engineering  development 
and  production  procurements . 

-  Chapter  2.  R&D  Acquisition  Process 

2- 13.  Technical  and  Business  Evaluations. 

b.  While  cost  or  price  should  not  be  the  controlling  factor  in 
selecting  a  contractor  for  an  R&D  contract,  they  should  not  be  disregarded 
in  the  choice  of  the  contractor. 
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APPENDIX  D 

COST-BENEFIT  ANALYSIS 


A.  INTRODUCTION 

The  purpose  of  this  Appendix  is  to  analyze  the  costs  and 
benefits  associated  with  applying  the  methodology  described  in 
this  Handbook.  In  particular  the  costs  and  benefits  of  making 
cost  realism  an  explicit  evaluation  criterion  are  addressed. 
Neither  the  costs  nor  the  benefits  can  be  readily  measured.  The 
principal  costs  associated  with  applying  the  methodology  are  the 
costs  of  preparing  detailed  Government  estimates  for  each 
offeror.  However,  even  if  the  methodology  is  not  followed 
completely,  some  examination  and  evaluation  of  each  offeror’s 
proposed  costs  must  be  made.  How  much  of  the  cost  associated 
with  proposal  cost  evaluation  is  related  to  cost  realism  or  a 
full  application  of  the  methodology?  The  principal  benefits 
associated  with  applying  the  methodology  are  possible  reductions 
in  cost  growth  and  schedule  slippage.  But  how  are  such  benefits 
quantified?  Although  exact  answers  to  these  questions  are 
difficult  to  determine,  the  following  subsections  try  to  place 
the  costs  and  benefits  into  perspective. 

B.  COST  REALISM  COSTS 

In  a  sense,  all  cost  evaluations  will  address  the  realism 
or  reasonableness  of  contractors’  offers.  However,  if  cost 
realism  is  made  an  explicit  evaluation  criterion,  in  an  effort 
to  encourage  the  contractors  to  make  more  realistic  offers,  then 
a  more  detailed  cost  evaluation  is  likely  than  otherwise.  The 
key  to  assessing  the  realism  and  reasonableness  of  cost 
proposals  is  the  development  of  a  realistic  Government  cost 
estimate  for  each  offeror,  which  can  be  compared  against  that 
offeror’s  proposed  cost.  This  is  termed  the  "Government 
estimated  cost"  or  "Government  estimate"  in  this  Handbook. 
While  a  cost  proposal  can  be  analyzed  and  its  realism  judged  on 
a  piecemeal  basis  (e.g.,  examining  a  few  major  cost  elements  and 
commenting  on  the  degree  of  realism),  such  an  approach  is 


generally  not  sufficient.  The  Government  should  make  an 
estimate  of  what  it  considers  to  be  the  most  likely  total  cost 
for  each  offeror.  Then  the  Government  estimates  can  be  compared 
to  the  offeror's  proposed  costs  to  determine  the  overall  degree 
of  cost  realism  and  the  Government  estimates  for  each  offeror 
can  be  compared  to  each  other  in  order  to  rank  the  offerors  in 
terms  of  the  Government  estimated  cost.  Therefore,  the  Govern¬ 
ment  estimates  themselves  must  be  realistic  and  reasonable. 

Developing  Government  estimates  for  each  offeror  requires  a 
detailed  analysis  of  direct  and  indirect  costs.  Direct  costs 
are  direct  labor,  direct  material,  and  other  direct  costs  which 
are  specifically  and  uniquely  attributable  to  a  particular 
product  or  service.  Direct  costs  are  the  "nucleus"  around  which 
total  contract  price  is  built  since  the  indirect  costs  (over¬ 
head,  G&A  and  fee)  are  generally  a  function  of  direct  costs. 

A  methodology  recommended  for  developing  these  estimates  is 
as  follows.  The  first  step  is  an  examination  of  the  cost 
proposals  and  any  backup  data  for  completeness  and  consistency. 
Then,  extensive  "fact-finding"  is  conducted  at  the  offerors' 
plants,  where  clarifications  can  be  obtained,  additional  backup 
data  examined,  and  the  offerors  asked  to  justify  their  cost 
estimates  based  on  their  actual  experience  on  previous  programs. 
Through  such  fact-finding,  the  validity  and  applicability  of  the 
offeror's  data  and  estimating  techniques  can  be  determined. 
Alternatively,  the  offerors  may  be  invited  to  Government  offices 
to  answer  questions;  but,  without  access  to  their  complete  data 
bases,  the  offerors  are  limited  in  the  answers  they  can  provide. 

A  number  of  costs  are  associated  with  source  selection  cost 
evaluation.  These  costs  are  primarily  a  function  of  the  time 
spent  by  Government  personnel.  Each  source  selection  organi¬ 
zation  includes  a  cost  evaluation  team  that  consists  of  several 
to  a  dozen  or  more  personnel  (generally  cost  and  pricing 
analysts)  depending  on  the  size  and  complexity  of  the  procure¬ 
ment,  the  number  of  offerors,  and  the  depth  of  the  cost  analy¬ 
sis.  These  personnel  are  generally  involved  in  the  source 
selection  nearly  full-time  for  a  period  of  one  to  a  few  months. 
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Therefore,  the  cost  evaluation  team  may  spend  from  a  few  man- 
months  to  50  or  more  man-months.  The  cost  evaluation  team 
typically  assists  in  preparation  of  the  Source  Selection  Plan 
and  the  solicitation,  prepares  a  detailed  Government  estimate 
which  may  include  detailed  fact-finding  at  contractors’  plants, 
evaluates  and  scores  the  costs,  and  prepares  appropriate  docu¬ 
mentation  of  their  cost  estimates  and  evaluation.  Obviously, 
the  cost  evaluation  can  be  done  at  many  levels  of  detail  and 
will  generally  be  more  detailed  if  cost  realism  is  an  evaluation 
criterion . 

The  amount  of  effort  expended  by  cost  evaluation  teams 
indicated  above  of  a  few  to  50  or  more  man-months  is  a  very 
rough  range.  Information  obtained  from  the  Cost  Analysis/ 
Estimating  Branch  of  NAVELEX  (ELEX-82B)  provides  a  better 
indication,  although  still  rough.  ELEX-82B  has  provided  a  major 
portion  of  the  cost  evaluation  teams  for  a  number  of  source 
selections.  These  source  selections  have  included  three  or  four 
RDT&E  source  selections  per  year  for  which  cost  realism  was  a 
specific  evaluation  criterion.  The  ELEX-82B  staff  of  9  to  12 
people  has  spent  approximately  one-third  to  one-half  of  their 
time  on  these  cost  evaluations.  Thus,  on  an  average,  each  cost 
evaluation  has  involved  somewhere  between  10  and  20  man-months. 
At  roughly  $4,000  per  man-month,  this  is  between  $40,000  and 
$80,000.  Travel  associated  with  the  fact-finding  increases 
these  costs  by  about  10  percent.  Since  the  average  contract 
value  is  about  $40  million,  the  average  cost  for  each  cost 
evaluation  is  between  0.1  percent  and  0.2  percent  of  the 
contract  value.  For  two  of  the  most  significant  RDT&E  cost 
evaluation  efforts,  ELEX-82B  has  made  the  following  estimates: 
18  man-months  for  the  ASPJ  and  24  to  27  man-months  for  the 
JTIDS.  In  each  of  these  cases,  the  RDT&E  contract  was  for  about 
$80  million.  Based  on  $4,000  per  man-month,  the  costs 
associated  with  these  cost  evaluations  were  $70,000  and 
$110,000,  respectively.  In  addition,  travel  costs  associated 
with  the  fact-finding  were  roughly  $10,000  for  each.  Therefore, 
the  cost  evaluation  team  costs  totaled  approximately  $80,000  and 


$120,000  or  0.10  percent  of  the  ASPJ  contract  value  and  0.15 
percent  of  the  JTIDS  contract  value. 

There  are  other  costs  than  the  cost  evaluation  team's  time 
and  travel  associated  with  source  selection  evaluation.  The 
team  is  assisted  by  various  organizations  including  DCAS,  DCAA 
and  related  organizations.  The  source  selection  technical  eval¬ 
uation  team  may  also  assist  the  cost  evaluation  team.  In  addi¬ 
tion,  the  offerors  may  spend  time  answering  questions  and 
supplying  additional  data,  particularly  during  the  fact-finding 
meetings.  In  some  cases,  these  other  costs  will  be  significant, 
and  may  even  double  the  costs  mentioned  above  if  DCAS  and/or 
DCAA  are  heavily  involved  in  the  cost  evaluation. 

The  NAVELEX  data  therefore  suggest  that  the  costs 
associated  with  RDT&E  cost  evaluations  for  electronic  systems, 
including  DCAS,  DCAA  and  other  support,  are  typically  0.1  to  0.3 
percent  of  the  contract  value.  This  includes  the  preparation  of 
a  detailed  Government  estimate  for  each  offeror,  fact-finding, 
and  updating  the  estimate  after  be s t-an d- f ina 1  offers  are 
received.  Data  were  not  available  to  determine  whether  the 
costs  of  aircraft,  missile  and  ship  cost  evaluations  are 
similar.  These  systems  generally  have  larger  RDT&E  costs  than 
electronic  systems,  and  there  is  generally  a  lot  more  that  has 
to  be  evaluated;  but  "economies-of-scale"  probably  come  into 
play  and  fewer  offerors  may  be  involved.  Therefore,  it  seems 
reasonable  to  assume  that  cost  evaluations  for  larger  systems 
will  not  exceed  0.1  to  0.3  percent  of  the  contract  value. 

The  above  discussion  provides  some  insights  into  the  costs 
associated  with  RDT&E  source  selection  cost  evaluations.  How¬ 
ever,  it  is  difficult  to  determine  how  much  of  this  cost  is 
associated  with  cost  realism.  Where  cost  realism  is  made  an 
explicit  evaluation  criterion,  as  in  the  NAVELEX  source  selec¬ 
tions  discussed  above,  a  more  intense  cost  evaluation  is  probab¬ 
ly  performed.  Certainly,  some  examination  and  evaluation  of 
each  offeror's  proposed  costs  must  be  made,  even  if  cost  realism 
is  not  an  evaluation  criterion.  Therefore,  only  a  fraction  of 
the  cost  evaluation  team  costs  is  attributable  to  cost  realism. 
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C.  COST  REALISM  BENEFITS 


A  major  reason  for  the  concern  about  cost  realism  is  that 
it  is  undoubtedly  a  significant,  but  sometimes  overlooked, 
factor  in  cost  growth.  Defense  acquisition  programs  have  long 
shown  tremendous  cost  growth  between  their  inception  and  field¬ 
ing.  Congress  has  expressed  great  interest  in  the  subject  of 
cost  growth  and,  in  November  1981,  took  the  unprecedented  step 
of  requiring  that  Service  Secretaries  prepare  a  detailed  report 
to  Congress  if  cost  growth  on  a  major  program  exceeds  15  percent 
or  face  suspension  of  further  obligations.  If  cost  growth 
exceeds  25  percent,  the  Secretary  of  Defense  must  certify  to 
Congress  that  the  program  is  essential  and  that  the  new  cost 
estimates  are  reasonable.* 

The  complexity  and  risks  inherent  in  Defense  systems  make 
contractor  cost  estimation,  not  to  mention  Government  review  and 
evaluation  of  the  costs,  difficult.  Some  optimism  may  be 
warranted  (e.g.,  in  engineering  hours  required,  in  number  of 
test  articles,  and  in  number  of  tests),  but  when  does  optimism 
become  unrealism?  In  some  cases,  the  contractor  may  purposely 
propose  an  unrealistically  low  cost  in  order  to  "buy-in"  and 
thereby  improve  its  business  base  and  cash  flow. 

Unrealistically  low  cost  estimates,  both  Government  and 
contractor,  contain  the  seeds  for  future  cost  growth.  Low  cost 
estimates  may  lead  to  u n d e r - b ud g e t i n g  and  to  schedule  slips 
which  in  turn  may  introduce  even  greater  costs  than  would  have 
occurred  with  realistic  budgets. 

Cost  growth  means  more  than  higher  costs  and  possible 
schedule  slips;  it  is  also  embarrassing,  a  factor  the  Nunn- 
McCurdy  Amendment  highlights.  Cost  growth  may  require  drastic 
changes  in  system  design  and  program  scope.  It  may  lead  to 
program  cancellation  or  to  the  cancellation  of  other  programs  in 
older  to  provide  the  required  additional  funds.  And  certainly, 
frequent  cost  overruns  will  undermine  and  eventually  destroy 

*  The  Nunn-McCurdy  Amendment  to  the  1983  DOD  Authorization 

Act . 
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public  confidence  in  the  Government's  capability  to  manage  its 
programs . 

Of  course,  the  Government  must  also  guard  against  unrealis¬ 
tically  high  cost  estimates.  This  situation  will  most  probably 
occur  if  there  are  few  technically  qualified  contractors  and  if 
a  contractor  already  has  an  especially  good  business  base.  High 
cost  estimates  do  minimize  the  likelihood  of  an  overrun,  but  the 
Government  could  probably  have  bought  the  system  for  less. 

Some  indication  of  the  degree  of  cost  realism  in  contrac¬ 
tors'  cost  proposals  and  the  possible  extent  to  which  the  cost 
realism  methodology  results  in  more  realistic  offers  is  provided 
by  data  collected  from  NAVELEX  on  proposed  costs  for  eight 
programs  and  the  corresponding  Government  estimates.  In 
developing  the  Government  estimates,  NAVELEX  evaluated  the 
contractor  cost  proposals  in  detail,  required  the  contractors  to 
justify  their  data  and  estimating  rationale,  developed  their  own 
estimates  based  almost  entirely  on  the  contractors'  data,  and 
compared  these  estimates  with  the  cost  proposals.  The  data 
showed  that  one  out  of  three  original  offers  was  more  than  20 
percent  low,  and  one  out  of  six  best-and-f ina  1  offers  was  more 
than  20  percent  low.  Offers  more  than  20  percent  below  the 
NAVELEX  estimate  certainly  raise  a  question  as  to  cost  realism. 
Interestingly,  less  than  one-quarter  of  the  original  offers  but 
nearly  half  of  the  best-and-f ina 1  offers  were  within  3  percent 
of  the  NAVELEX  estimates. 

In  summary,  then,  if  greater  cost  realism  can  be  achieved 
in  contractors'  cost  proposals  through  application  of  the 
methodology  described  in  this  Handbook,  or  through  any  other 
means,  some  of  the  likely  benefits  (not  necessarily  mutually 
exc lusive)  are: 

-Reduced  cost  growth, 

-Reduced  incidence  of  "buy-ins", 

-Reduced  under-budgeting, 

-Reduced  schedule  slippages, 

-Reduced  program  turbulence  resulting  from  the  above 
factors , 
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-Reduced  possibility  of  major  redesign,  scope  changes 
and  program  cancellations, 

-Fewer  reports  to  Congress  under  the  Nunn-McCurdy 
Amendment,  and 

-Reduced  costs  in  situations  where  an  offer  is 
unrealistically  high,  but  due  to  other  considerations 
that  offeror  wins  the  competition. 

This  list  of  benefits  is  not  meant  to  imply  that  cost 
realism  is  a  magic  elixir,  but  these  benefits  can  be  achieved  in 
some  measure.  After  all,  contractors’  cost  proposals  can  often 
be  more  realistic  than  they  are  now;  in  many  cases,  both  the 
data  base  and  the  tools  exist  for  contractors  to  make  more 
realistic  estimates. 

The  cost  realism  benefits  mentioned  above  are  not  easily 
quantified.  However,  it  is  possible  to  give  some  notion  of  the 
possible  reduction  in  cost  growth  and  the  dollar  value  of 
reduced  schedule  slippage. 

First,  the  possible  reduction  in  cost  growth  is  discussed. 
The  improvements  between  initial  offers  and  best-and-f ina 1 
offers  noted  in  the  NAVELEX  data  presented  above  were  apparently 
largely  the  result  of  the  detailed  Government  cost  evaluation 
and  the  fact  that  cost  realism  was  an  evaluation  criterion.  Low 
initial  offers  came  up  an  average  of  9  percent  in  the  best-and- 
final  offers  (e.g.,  from  20  percent  below  the  Government 
estimate  to  11  percent  below).  While  this  result  is  not  entire¬ 
ly  due  to  the  application  of  a  cost  realism  methodology  by 
NAVELEX,  the  cost  realism  methodology  was  undoubtedly  a  major 
factor.  Application  of  the  cost  realism  methodology  gives 
offerors  an  incentive  to  be  more  realistic  through  the  scoring 
mechanism  and  the  fact  that  very  unrealistic  offers  may  be  the 
basis  for  disqualification.  These  data  as  well  as  intuition 
suggest  that  the  cost  realism  methodology  may  result  in  offers 
(at  least  b e s t - and - f i n a  1  offers)  which  are  more  realistic,  as 
measured  by  the  Government  estimates,  by  10  percent  and  even 
considerably  more. 
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Now  the  possible  reduction  in  program  costs  due  to  reduced 
schedule  slippage  is  discussed.  Suppose  a  five  year  development 
program  stretches  to  a  six  year  program  due  to  funding  problems 
and  program  turbulence  created  by  significant  cost  growth. 
Certain  program  costs  are  of  a  fixed  or  level-of-ef f ort  nature, 
such  as  program  management  costs,  and  are  more  a  function  of 
time  than  a  function  of  the  technical  tasks  to  be  performed. 
Assume  that  the  RDT&E  program  is  estimated  to  cost  $20  million 
per  year  for  five  years  — a  total  of  $100  mil  lion--and  that  10 
percent  or  $2  million  per  year  is  fixed.  A  program  stretch  to 
six  years  will  add  $2  million  to  the  total  program  cost  —  even 
more  if  the  variable  (non-fixed)  costs  cannot  be  easily 
rescheduled  (e.g.,  certain  engineers  might  have  to  be  kept  on 
the  program  longer  or  alternatively  might  have  to  be  terminated 
and  replacements  hired).  The  added  $2  million  due  to  the  one 
year  schedule  slip  adds  two  percent  to  the  total  program  cost. 
A  six  month  stretch  would  add  one  percent.  This  probably  under¬ 
states  the  impact  of  schedule  slips  since  many  activities  and 
personnel  cannot  be  easily  rescheduled  as  suggested  above. 
Schedule  slips  can  be  expensive  and  unrealism  in  costs  can 
certainly  lead  to  schedule  slips.  A  contract  award  to  an 
offeror  who  is  unrealistic  by  20  percent  compared  to  a  Govern¬ 
ment  estimate  could  very  well  result  in  a  program  having  only  80 
percent  of  the  budget  it  needs  and  a  program  slip  of  a  year  or 
so  by  the  time  additional  funding  is  made  available.  Therefore, 
in  a  situation  like  this,  if  the  offeror  is  encouraged  to  make  a 
more  realistic  offer,  it  is  reasonable  to  assume  that  several 
percent  of  the  program  cost*  can  be  saved. 

D.  CONCLUSIONS 

A  methodology  has  been  developed  for  assuring  greater  cost 
realism  in  contractors'  RDT&E  proposals,  based  largely  on  tech¬ 
niques  and  procedures  developed  by  NAVELEX.  Key  aspects  of  this 

*  The  estimated  program  cost  if  there  had  net  been  a 
schedule  slip. 
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methodology  are  (a)  establishing  cost  realism  as  an  explicit 
cost  evaluation  criterion  in  the  solicitation,  (b)  preparing  a 
detailed  Government  estimate  for  each  offeror  based  largely  on 
the  offeror’s  own  data,  and  (c)  scoring  the  degree  of  cost 
realism.  While  the  costs  and  benefits  associated  with  applying 
this  methodology  are  difficult  to  quantify,  this  appendix  has 
tried  to  place  the  costs  and  benefits  into  perspective. 
Unrealism  in  cost  proposals  often  contributes  to  cost  growth, 
which  may  lead  to  program  turbulence  and  schedule  slips. 
Program  costs  may  increase  by  at  least  several  percent  over  what 
the  program  would  have  cost  had  an  award  been  made  at  a 
realistic  cost  and  budgeted  accordingly.  On  the  other  hand,  the 
costs  associated  with  source  selection  cost  evaluations,  only 
part  of  which  can  be  specifically  attributed  to  the  cost  realism 
evaluation  criterion,  are  probably  less  than  a  few  tenths  of  a 
percent  of  the  program  cost.*  Therefore,  on  the  basis  of  this 
rough  cost-benefit  analysis  as  well  as  the  intangible  benefits 
which  have  been  identified,  there  appears  to  be  ample  justifi¬ 
cation  for  greater  attention  to  cost  realism  in  the  source 
selection  process. 

If  desired,  a  somewhat  less  rough  cost-benefit  analysis  can 
be  performed  than  was  possible  in  this  Appendix.  For  example, 
the  size  and  schedule  of  the  cost  evaluation  team  will  be  known 
or  can  be  estimated  for  the  specific  program.  Also,  an  approxi¬ 
mate  cost  of  the  specific  program  will  be  known  or  can  be 
estimated.  Further,  it  may  be  possible  to  determine  approxi¬ 
mately  how  a  contract  award  well  below  the  likely  cost  of  the 
program  will  affect  the  program  budget  and  schedule  as  well  as 
what  impact  a  schedule  slip  would  have  on  1 e ve 1-of-ef f or t  costs. 
However,  even  without  performing  a  cost-benefit  analysis  for 


*  The  costs  associated  with  a  cost  evaluation  are  incurred 
during  the  source  selection  process  whereas  any  cost  savings 
attributable  to  cost  realism  would  be  realized  during  the  course 
of  the  RDT&E  contract.  Therefore,  to  be  more  accurate,  the  cost 
savings  should  be  discounted.  Since  the  quantification  of  the 
costs  and  benefits  in  this  report  is  very  approximate,  no 
discounting  was  done. 
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each  specific  source  selection,  there  appears  to  be  ample  justi¬ 
fication,  tangible  and  intangible,  for  applying  the  methodology 
in  this  Handbook. 
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APPENDIX  E 

NORMAL  CURVE  SCORING  RULE 


As  mentioned  in  Section  VIII,  various  smooth  curves  can  be 
used  as  scoring  rules.  A  normal  curve  is  a  particularly 
convenient  form  of  a  smooth  curve.  The  parameters  of  a  normal 
curve  can  be  chosen  such  that  a  normal  curve  scoring  rule  close¬ 
ly  approximates  a  triangle  or  trapezoid  scoring  rule.  The 
advantage  of  a  normal  curve  scoring  rule  is  that  it  does  not 
have  sharp  discontinuities  as  do  the  the  triangle  and  trapezoid. 
However,  the  normal  curve  scoring  rule  is  not  as  easy  to  explain 
and  understand.  Furthermore,  a  normal  curve  is  generally  asso¬ 
ciated  with  statistical  probabilities  or  with  uncer tainties-- 
connotations  which  are  not  appropriate  in  this  case.  Therefore, 
the  discussion  in  Section  VIII  assumed  a  triangle  or  trapezoid 
scoring  rule.  Since  some  readers  of  this  Handbook  may  prefer 
the  normal  curve  scoring  rule,  this  Appendix  provides  a  discus¬ 
sion. 

Normal  curve  scoring  rules  are  discussed  below  first  for 
cost  realism  and  then  for  Government  estimated  cost. 

A.  COST  REALISM 

The  normal  curve  scoring  rule  for  cost  realism  is  a  "bell" 
shaped  curve  whose  "spread"  is  determined  by  the  normal  curve 
standard  deviation,  (7,  as  shown  below: 


D'  is  the  offer  (C)  relative  to  the  Government  estimated  cost 
(G)  in  percentage  terms  or  (C-G)/G  %.  There  is  a  68  percent 


E-l 


likelihood  that  D'  will  be  less  than  a  ,  and  a  95  percent  likeli¬ 
hood  that  D'  will  be  less  than  2 a  (i.e.,  68  percent  of  the  area 
under  the  curve  is  within  O  of  the  mode,  and  95  percent  of  the 
area  is  within  2 o  ).  The  standard  deviation  is  used  here  to 
define  the  spread  of  the  scoring  rule  and  does  not  necessarily 
have  a  statistical  significance. 

In  order  to  use  the  normal  curve  scoring  rule  for  cost 
realism,  two  values  must  be  determined — the  maximum  score  and 
the  "spread".  It  is  convenient  to  set  the  maximum  score  at  10. 
For  triangle  and  trapezoid  scoring  rules,  the  spread  is  defined 
as  the  point  at  which  the  score  becomes  a  minimum.  For  the 
normal  curve,  the  spread  can  conveniently  be  defined  as  2a.  For 
example,  if  20  =  20  percent  of  G,  then  an  offer  10  percent 
higher  or  lower  than  G  would  receive  a  score  of  6.1  and  an  offer 
20  percent  higher  or  lower  would  receive  a  score  of  1.4.  This 
is  illustrated  as  follows: 


The  equation  for  the  scoring  rule  is: 


= 


/?rr2 


n  i  2  / o~2 


By  coincidence,  normal  curve  and  trapezoid  scoring  rules 
with  the  same  spread  (2 O  =  S)  closely  approximate  each  other 
(assuming  the  trapezoid  plateau  is  0.1S  and  the  minimum  score  is 
1  at  S).  This  is  illustrated  in  Figure  E.l. 

The  normal  curve  scoring  rule  has  th  following  desirable 
properties.  First,  it  is  a  smooth  function  which  can  be  expres¬ 
sed  by  an  exponential  equation  as  given  above.  Once  plotted  for 
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any  value  of  2 a,  the  normal  curve  is  easy  to  use;  just  figure  D' 
and  then  read  off  the  score. 

Second,  offers  close  to  the  Government  estimate  have 
approximately  the  same  high  score  and  offers  far  from  the 
Government  estimate  have  approximately  the  same  low  score. 

Third,  and  most  importantly,  since  the  spread  (2  0  or  S)  is 
arbitrarily  chosen  for  any  program  and  a  greater  or  lesser 
spread  may  in  fact  be  more  reasonable,  it  is  desirable  that,  as 
the  spread  changes,  the  scores  change  with  a  minimum  of  distor¬ 
tion.  The  triangle  and  trapezoid  scoring  rule  can  lead  to 
greater  distortions  than  the  normal  curve  scoring  rule.  This  is 
indicated  for  the  trapezoid  in  the  table  on  the  following  page. 
For  example,  as  a  result  of  a  change  in  S  from  15  to  20  percent, 
the  second  offeror's  score  increases  by  2.5  points  relative  to 
the  third  offeror,  if  the  trapezoid  scoring  rule  is  used.  How¬ 
ever,  if  the  normal  curve  scoring  rule  is  used,  as  2 o  changes 
from  15  to  20  percent,  the  second  offeror's  score  increases  by 
only  0.8  points  relative  to  the  third  offeror.  Figure  E.2 
further  illustrates  this  problem  with  the  trapezoid  scoring 
r  uie . 


B.  GOVERNMENT  ESTIMATED  COST 

A  Government  estimated  cost  scoring  rule  using  part  of  a 
normal  curve  is  illustrated  below: 


COST  RELATIVE  TO  G* 


D  is  the  Government  estimated  cost  (G)  relative  to  She  lowest 
Government  estimated  cost  (G*)  in  percentage  terms  or  (G-G*)/G* 
%.  In  order  to  more  closely  approximate  a  triangle  scoring 
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Figure  E.2  Differences  in  Scores  for  Different  Spreads 


SCORES  FOR  DIFFERENT  VALUES  OF  S  AND  2a 


Offeror 


A 

B 

C 

D 

10% 

15% 

20% 

25% 

Trapezoid* 

S  =  15% 

S  =  20% 
Difference 


rm 

2o 

2o 


Difference 


rule,  this  normal  curve  scoring  rule  has  been  shifted  as 
illustated  below: 


0  ,5a  2a  UNSHIFTED 

- . 33S  0  S  SHIFTED 

COST  RELATIVE  TO  G*  {%) - ► 


To  use  the  normal  curve  scoring  rule  for  Government 
estimated  cost,  two  values  must  be  determined--the  maximum  score 
and  the  "spread".  (The  amount  of  the  shift  could  also  be 
varied.)  As  for  cost  realism,  it  is  convenient  to  set  the 
maximum  score  at  10.  The  spread,  S,  is  defined  for  convenicence 
as  2  (see  the  above  illustration).  For  example,  if  S  =  40 
percent  of  G*,  then  an  offer  20  percent  higher  than  G*  would 
receive  a  score  of  5.2  and  an  offer  40  percent  higher  would 
receive  a  score  of  1.5.  This  is  illustrated  as  follows: 
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COST 


(D  -  ) 


The  equation  for  the  scoring  rule  is: 


Score  =  11.33e-'^%  f)2/0.889S*  .  n.33,-<D4>*/0.889S» 

A  normal  curve  and  triangle  scoring  rule  with  the  same 
spread  closely  approximate  each  other.  This  is  illustrated  in 
Figure  E.3. 

The  normal  curve  scoring  rule  has  the  same  advantages  for 
Gvoernment  estimated  cost  as  for  cost  realism. 
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APPENDIX  F 

COST  REALISM  COMPUTER  MODELS 


A.  INTRODUCTION 

Two  computer  models  have  been  developed  to  assist  source 
selection  personnel  in  evaluating  cost  proposals.  The  first 
model  is  a  "cost  evaluation  scoring  model"  for  generating 
numerical  scores  for  cost  proposals.  The  second  model  is  a 
"cost  evaluation  tradeoff  model"  for  determining  the  tradeoffs 
between  cost  evaluation  scores  and  technical  scores.  Both 
models,  but  especially  the  latter  one,  can  assist  source 
selection  personnel  in  establishing  the  shape  of  cost  evaluation 
scoring  rules  and  the  weights  to  be  assigned  to  cost  evaluation 
criteria. 

The  scoring  methodology  that  underlies  these  models  is 
explained  in  Section  VIII  of  this  Handbook.  Two  versions  of 
each  model  have  been  developed;  one  version  is  based  on  a  trape¬ 
zoid  shaped  scoring  rule,  and  the  other  version  is  based  on  a 
normal  curve  scoring  rule.  (Trapezoid  shape  as  used  in  this 
Appendix  also  includes  triangle  shape.)  The  two  versions 
produce  very  similar  results.  The  following  discussion  applies 
to  both  versions,  except  that  the  "top"  spread  and  minimum  score 
apply  only  to  the  trapezoid  version.  Tables  and  figures  follow 
the  text  and  are  based  on  the  trapezoid  version  since  it  is 
easier  to  understand.  However,  examples  of  the  normal  curve 
version  are  presented  in  Subsection  D. 

B.  SCORING  MODEL 
Main  Scoring  Model 

An  illustrative  scoring  model  run  is  shown  in  Table  1  and 
corresponds  to  the  illustrative  example  in  Section  X.  Boxes  are 
drawn  around  the  inputs  that  are  required--on  the  screen,  the 
cells  requiring  inputs  are  highlighted.  Two  cost  evaluation 
criteria  are  a s s um ed --Go v e r n me n t  Estimated  Cost  and  Cost 
Realism.  The  Government  Estimated  Cost  is  evaluated  on  the 
basis  of  the  percent  difference  between  the  Government  Estimated 
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Cost  for  a  particular  offeror  and  the  lowest  Government  estimate 
for  all  of  the  competing  offerors  (Delta).  Cost  Realism  is 
evaluated  on  the  basis  of  the  percent  difference  between  each 
offer  and  the  Government  Estimated  Cost  for  that  offeror 
(Delta').  The  "base"  spread  is  the  Delta  or  Delta'  at  which  the 
scoring  rule  assigns  a  minimum  score.  The  "top"  spread  is 
typically  10%  of  the  base  spread;  and  the  minimum  score  is 
typically  "1".  A  choice  is  given  of  entering  the  weights  or 
weight  ratios;  whichever  is  entered,  the  other  is  automatically 
calculated . 

Once  the  spreads,  weights,  offers  and  Government  estimates 
are  entered,  then  Delta,  Delta',  cost  evaltuation  scores  and 
weighted  scores  are  calculated,  as  illustrated  in  Table  1.  The 
degree  of  technical  advantage  that  an  offeror  would  have  to  have 
over  the  offeror  with  the  highest  cost  evaluation  weighted  score 
in  order  to  equalize  their  total  weighted  scores  is  also  shown 
(the  range  results  from  the  assumption  that  the  technical  scores 
may  be  between  70  and  100%  of  the  maximum  technical  score).  The 
sensitivity  of  the  scores  to  a  difference  of  plus  or  minus  2%  in 
the  Government  estimate  is  also  shown. 

When  the  scoring  model  is  loaded,  a  "blank"  table  is  dis¬ 
played  with  zeros  or  "errors"  shown  for  the  input  values.  The 
illustrative  example  in  Table  1  may  be  quickly  loaded  into  the 
model  as  indicated  on  the  second  page  of  the  t-'ible.  Six 
different  graphs  of  the  scores  may  be  quickly  displayed  on  the 
screen.  The  graphs  for  the  above  illustrative  example  are  dis¬ 
played  in  Figures  1  to  6.  Instructions  for  printing  the  table 
and  the  graphs  are  provided  on  the  second  page  of  the  table. 

Sensitivity  Scoring  Model 

The  above  table  and  graphs  represent  the  "main"  scoring 
model  [ SCTRAP  ] .  There  is  a  second  part  to  the  scoring  model 
[SCTRAP/2]  which  provides  a  sensitivity  analysis  of  the  spreads 
and  weights.  Two  different  spreads  and  two  different  weights 
(or  weight  ratios)  are  entered  into  the  "sensitivity"  table. 
Three  different  graphs,  showing  the  sensitivities,  may  be 
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quickly  displayed.  This  part  of  the  model  is  illustrated  in 
Table  2  and  Figures  7  to  9. 

Comparison  Scoring  Model 

There  is  also  a  third  part  to  the  scoring  model  [SCCOMPT] 
which  provides  a  comparison  of  scores  for  three  cases  with 
different  spreads  and  weights.  Note  that  the  second  and  third 
parts  of  the  model  are  different  ways  to  look  at  the  effect  of 
spreads  and  weights  on  the  scores.  The  three  cases  are  entered 
into  the  "comparison"  table.  Two  different  graphs  may  be 
quickly  displayed.  The  comparison  part  of  the  model  is 
illustrated  in  Table  3  and  Figures  10  and  11.  Note  that  this 
illustration  has  the  same  weights  in  all  three  cases.  Another 
illustration  of  the  comparion  model  with  different  weights,  but 
with  the  spreads  the  same  in  all  three  cases,  is  provided  in 
Table  4  and  Figures  12  and  13.  These  two  runs  of  the  comparison 
model  illustrate  the  effect  on  scores  of  a  broad  range  of  values 
for  the  spreads  and  weights. 

Studying  Tables  3  and  4  and  their  associated  graphs 
(together  with  the  illustrative  tradeoff  model  runs  discussed  in 
Section  C)  can  provide  a  lot  of  help  in  establishing  spreads  and 
weights  to  be  used  for  a  particular  source  selection 
e v a  1 uation--without  necessarily  having  to  actually  run  the 
model. 

It  should  be  noted  that  the  the  scoring  model  can  only 
accomodate  five  offerors.  If  there  are  more  than  five  offerors, 
additional  runs  of  the  scoring  model  will  have  to  be  made.  The 
offeror  with  the  lowest  Government  estimate  must  be  repeated  in 
all  runs.  Tables  and  graphs  with  all  offerors  can  then  be 
prepared  manually. 

C.  TRADEOFF  MODEL 

The  tradeoff  model  [TRDTRAP]  is  illustrated  in  Table  5. 
Boxes  are  drawn  around  the  inputs  that  are  required.  As  many  as 
three  sets  of  spreads  and  three  sets  of  weights  may  be  input  to 
the  model.  However,  a  single  set  is  sufficient  to  run  the 
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model.  The  most  challenging  inputs  required  in  the  table  are 
the  values  for  the  indifference  envelopes  (these  envelopes  are 
discussed  in  the  "Cost  Realism  Handbook").  The  indifference 
envelope  for  the  Government  Estimated  Cost  is  that  range  of 
costs  that  one  would  be  willing  to  accept  for  a  particular 
technical  advantage.  The  indifference  envelope  for  cost  realism 
is  that  range  of  unrealism  that  one  would  be  willing  to  accept 
for  a  particular  technical  advantage.  The  indifference 
envelopes  can  be  entered  as  percentages  or  dollars  for  three 
different  technical  advantages.  If  the  approximate  cost  of  the 
program  is  specified,  then  dollars  will  automatically  be  calcu¬ 
lated  if  percentages  are  entered,  and  vice  versa.  Once  the 
spreads,  weights,  and  envelopes  are  entered,  the  tradeoffs  are 
calculated . 

When  the  tradeoff  model  is  loaded,  a  "blank"  table  is 
displayed  with  zeros  or  "errors"  shown  for  the  input  values.  The 
illustrative  example  in  Table  5  may  be  quickly  loaded  into  the 
model  as  indicated  on  the  first  page  of  the  table.  Fourteen 
different  graphs  of  the  tradeoffs  may  be  quickly  displayed  on 
the  screen.  The  graphs  for  the  above  illustrative  example  are 
displayed  in  Figures  1A  to  27.  Note  that  starting  with  the 
"blank"  table,  just  the  illustrative  ind i f f ererence  envelopes 
may  be  quickly  loaded  into  the  model.  Instructions  for  printing 
the  table  and  the  graphs  are  provided  on  the  second  page  of  the 
table  . 

The  tradeoff  model  run  in  Table  5  illustrates  the  effect  on 
tradeoffs  of  a  broad  range  of  values  for  the  spreads  and 
weights  —  the  same  broad  range  used  for  the  comparison  scoring 
model  runs  in  Tables  3  and  A.  The  technical  weights  are  held 
constant  at  60  (out  of  100)  in  Table  5,  and  the  total  cost 
t  valuation  weights  (Government  Estimated  Cost  plus  Cost  Realism) 
are  held  constant  at  30.  Therefore,  a  further  illustration  of 
the  tradeoff  model  is  provided  in  Table  6  and  Figures  28  to  33 
with  the  technical  weights  varied  from  50  to  75  and  the  total 
cost  evaluation  weights  varied  from  20  to  35. 


Studying  Tables  5  and  6  and  their  associated  graphs, 
together  with  the  illustrative  scoring  model  runs  in  Tables  3 
and  4,  can  provide  a  lot  of  help  in  establishing  spreads  and 
weights  to  be  used  for  a  particular  source  selection  evaluation 
without  necessarily  having  to  actually  run  the  models. 

D.  MISCELLANEOUS 
Lotus  1-2-3 

The  scoring  model  and  tradeoff  model  have  been  developed 
using  the  Lotus  1-2-3  integrated  software  program.  The  Lotus 
program  is  one  of  the  most  popular  and  widely  used  spreadsheet/ 
graphics  programs.  It  runs  on  an  IBM  PC  with  196K  of  memory,  as 
well  as  a  number  of  other  micro-computers.  The  models  could  be 
readily  adapted  to  other  spreadsheet/graphics  programs  by  users, 
if  necessary.  Two  advantages  of  Lotus  and  similar  integrated 
programs  are  the  ease  of  displaying  graphs  and  the  ease  of 
making  modifications. 

To  use  one  of  the  models,  Lotus  1-2-3  is  loaded  into  the 
computer  along  with  a  data  disk  containing  the  model.  There  are 
separate  data  disks  for  the  scoring  and  tradeoff  models.  Copies 
of  these  special  data  disks  may  be  obtained  from  the  Navy  Office 
of  Acquistion  Research. 

Normal  Curve  Scoring  Rules 

As  mentioned  at  the  beginning,  two  versions  of  the  models 
have  been  developed — a  trapezoid  scoring  rule  version  which  has 
been  used  for  the  above  tables  and  figures  and  a  normal  curve 
scoring  rule  version.  The  scoring  rules  are  compared  in  Figures 
34  and  35.  The  two  versions  produce  very  similar  results. 
Normal  curve  scoring  rules  are  discussed  in  Appendix  E.  An 
illustrative  normal  curve  scoring  model  run  is  pre°  *■  e  d  in 
Table  7  and  Figures  36  and  37.  This  example  is  xcal  to 
that  presented  in  Table  1,  except  for  the  use  of  a  normal  curve. 
Comparisons  between  the  trapezoid  and  normal  scores  in  Table  1 
and  7  are  shown  in  Figures  38  and  39. 
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An  illustrative  normal  curve  tradeoff  model  run  is  pre¬ 
sented  in  Table  8  and  Figures  40  to  43.  This  example  is  identi¬ 
cal  to  that  presented  in  Table  5,  except  for  the  use  of  a  normal 
curve . 


E.  INFORMATION  FOR  USERS  OF  THE  MODEL 

The  several  models  and  their  names  are  shown  in  the  table 
below.  Associated  files  are  also  indicated. 


Model  Type 

Model  Name 

Data 

Disk 

Example 

File 

("ALT  X") 

Envelopes 

File 

("ALT  V") 

Blank 

File 

("ALT  Y") 

Trapezoid  Shape: 

Main  Scoring 

SCTRAP 

A 

XT 

— 

YT 

Sensitivity 

SCTRAP/2 

A 

UT 

- 

VT 

Comparison 

SCCOMPT 

A 

XTC 

— 

YTC 

Tradeoff 

TRDTRAP 

B 

XT 

V 

YT 

Normal  Curve  Shape: 

Main  Scoring 

SCNORM 

A 

X 

- 

Y 

Sensitivity 

SCN0RM/2 

A 

u 

- 

V 

Comparison 

SCCOMPN 

A 

xc 

— 

YC 

Tradeoff 

TRDNORM 

B 

X 

V 

Y 

The  models  are  loaded  as  follows.  With  the  Lotus  1-2-3 
disk  in  Drive  A  and  the  scoring  model  or  tradeoff  model  data 
disk  in  Drive  B,  boot  the  system.  If  the  system  is  already 
booted  and  the  system  prompt  "  A  >  "  is  displayed,  type  "123"  and 
"enter".  When  the  blank  spreadsheet  appears,  type  "/FR"  and 
select  the  appropriate  model  by  moving  the  cursor  over  the  model 
name  and  pressing  "enter".  Each  of  the  model  tables  provides 
most  of  the  information  needed  to  run  the  models.  The  key 
information  is  summarized  in  the  next  paragraph.  Additional 
information  is  provided  in  subsequent  paragraphs. 

When  using  the  models,  it  should  be  noted  that  after 
entering  or  changing  data  in  a  table  it  is  necessary  to  press 
the  "F9"  key  in  order  to  perform  a  recalculation.  DO  NOT 
FORGET!  Graphs  are  displayed  or  "invoked"  by  simultaneously 
pressing  the  "ALT"  key  and  a  special  letter  key,  as  explained  in 
each  input  table.  To  clear  a  graph  from  the  screen  and  return 
to  an  input  table,  it  is  necessary  to  press  a  key  (any  key). 


F-6 


When  changing  from  entering  weights  as  whole  numbers  to 
entering  as  ratios,  or  vice  versa,  first  return  to  a  "blank" 
table.  This  may  be  accomplished  using  "ALT"  Y,  which  replaces 
the  weight  computational  formulas. 

A  "circular"  error  will  sometimes  be  displayed  by  Lotus  1- 
2-3.  This  will  occur  with  a  "blank"  table  or  when  not  all  of 
the  weight  input  cells  are  filled-in  and  is  not  a  cause  for 
alarm . 

In  using  the  models,  there  may  be  occasion  to  save  a  table 
with  values  filled  in.  In  this  case  be  sure  to  save  the  table 
using  a  different  name. 

Also,  on  occasion  the  user  may  want  to  modify  the  model  in 
some  ways.  Changes  may  not  be  made,  except  for  the  highlighted 
input  values,  unless  the  cell  protection  feature  is  disabled  by 
using  "/WGPD".  Protection  may  be  turned  back  on  using  "/WGPE". 

A  listing  of  the  computer  model  data  input  cells  and  equa¬ 
tions  for  SCTRAP  and  TRDTRAP  are  provided  in  Tables  9  and  10. 
As  mentioned  earlier,  data  disks  for  use  with  LOTUS  1-2-3  may  be 
obtained  from  the  Navy  office  of  Acquisition  Research  (address 
on  cover). 
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Table  \ 


COST  EVALUATION  SCORING  MODEL — TRAPEZOID  SHAPE  CSCTRAP3 

ENTERt  SPREADS  AS  DECIMALS 


WEIGHTS  AS  WHOLE  NUMBERS  -OR-  ENTER  WEIGHT  RATIOS  AS  DECIMALS 

MIN 


SPREAD! 

BASE 

TOP 

SCORE 

GOV’T  EST 

COST 

40.  0* 

0.  054 

1 

3 

COST  REALISM 

20.  0% 

2.  054 

1 

3 

WEIGHT! 

TECHNICAL 

<T) 

60.0 

3 

GOV’T  EST 

COST  (E) 

20.  0 

3 

AFTER 

ENTERING 

OR  CHANGING 

DATA, 

COST  REALISM  <R> 

10.  0 

3 

PRESS 

"F9"  TO  RECALCULATE 

OTHER 

10.  0 

3 

— 

TOTAL 

1O0.  0 

1 

1 

or 

WEIGHT  RATIO! 

1 

1 

E/T 

33.  3 54 

3 

R/T 

16.  754 

3 

E/ (E+R) 

66.  7 54 

R/ (E+R) 

33.  354 

OFFER  i 

BOV'T  EST 

<G- 

-G*)  /G* 

(C-G)/G  • 

- SCORES - 

OFFEROR 

<C> 

(G) 

(DELTA) 

(DELTA’  )  GOV’ T  EST 

REALISM 

A 

47.  O 

48.0 

0.  054 

-2.  154 

10.0 

10.0 

B 

47.0 

54.0 

12.554 

-13.  054 

7.2 

4.5 

C 

53.0 

50.0 

4.  254 

6.  054 

9.  1 

8.0 

D 

60.  O 

65.0 

35.  454 

-7.  754 

2.0 

7.2 

E 

89.0 

70.0 

45.  854 

27.  154 

1.0 

1.0 

WEIGHTED  SCORES -  TECH  ADV  TRADEOFF 


OFFEROR 

GOV’T  EST  1 

REALISM 

TOTAL 

54  OF  MAX 

LOW 

HIGH 

A 

20.  0 

10.0 

30.0 

10054 

054 

054 

B 

14.4 

4.  5 

18.9 

6354 

2354 

2654 

C 

18.  1 

8.0 

26.  1 

8754 

7% 

954 

D 

4.  1 

7.  2 

11.2 

3754 

4554 

4554 

E 

2.  0 

1.0 

3.  0 

1054 

8254 

8254 

/  J.  /  o  V  \ 

i  n  nni5ii7E. 

- GOV’T  EST 

COST — 

- COST  REALISM - 

OFFEROR 

LOW 

HIGH 

LOW 

HIGH 

A 

0.  054 

2.  054 

-0.  154 

-4.  154 

B 

10.  554 

14.  554 

-1 1. 054 

-15.  054 

C 

2.  254 

6.  254 

4.  054 

8.  054 

D 

33.  454 

37.  454 

-5.  754 

-9.  754 

E 

43.  854 

47.  854 

25.  154 

29.  1% 

crnoc 

DAKIRC 

LictruTcn  i 

OwU  “t 

rCHrMof” 

wc.  a  on  i  ouunc 

- GOV’T  EST 

COST — 

- COST  REALISM - 

TOTAL 

TOTAL 

OFFEROR 

LOW 

HIGH 

LOW 

HIGH 

LOW 

HIGH 

A 

9.6 

10.0 

9.0 

10.0 

28.  1 

30.0 

B 

6.  7 

7.6 

3.  5 

5.5 

17.0 

20.8 

C 

8.6 

9.5 

7.0 

9.0 

24.2 

28.0 

D 

1.6 

2.5 

6.  2 

8.2 

9.  3 

13.  1 

E 

1.0 

1.0 

1.0 

1.0 

3.  0 

3.0 

Table  1  (con't) 

QUICK  ENTRIES! 


TO  LOAD  COMPLETE  EXAMPLE,  PRESS  "ALT”  KEY  AND  "X"  SIMULTANEOUSLY 
TO  RETURN  TO  BLANK  ENTRY  FORM,  PRESS  “ALT"  KEY  AND  “Y"  SIMULTANEOUSLY 

INVOKE  GRAPHS!  PRESS  "ALT"  KEY  AND  INDICATED  LETTER  SIMULTANEOUSLY 

-  (IF  THERE  IS  A  GRAPH  ON  THE  SCREEN,  FIRST  PRESS  ANY  KEY) 

(TO  RETURN  TO  THE  WORKSHEET,  PRESS  "Q"> 

GOV’T  EST  REALISM  WEIGHTED 
SCORE  ABC 

RANGE  D  E  F 

PRINT  TABLE!  PRESS  "ALT"  KEY  AND  "P"  SIMULTANEOUSLY 


PRINT  GRAPHS:  SAVE  EACH  GRAPH  TO  A  "PRINT-GRAPH"  FILE  AS  FOLLOWS! 


INVOKE  GRAPH  AS  ABOVE,  PRESS  ANY  KEY,  PRESS  "S",  AND 
TYPE  IN  A  "PRINT-GRAPH"  NAME  USING  APPROPRIATE  LETTER 
FROM  ABOVE  TABLE 

IF  MORE  THAN  ONE  GRAPH  IB  TO  BE  SAVED  OF  THE  SAME  TYPE 
(I.E.  ,  DIFFERENT  ENTRIES  IN  THE  TABLE),  USE  A  DIFFERENT 
NAME  FOR  EACH  GRAPH — FOR  EXAMPLE,  Al,  Bl,  ETC. 

THEN,  USE  THE  LOTUS  "PRINT-GRAPH"  DISK 

SENSITIVITY  ANALYSIS:  GO  TO  NEXT  PART 


Figure  1. 

GOVERNMENT  ESTIMATE  SCORE 


Figure  2. 

COST  REALISM  SCORE 
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Figure  3. 
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Table  2. 

SCORING  SENSITIVITY  TO  SPREADS  AND  WEIGHTS — TRAPEZOID  SHAPE  CSCTRAP/23 


ENTER  SPREADS  AND  WEIGHTS 


- BASE- 

1ST 

2ND 

- TOP- 

1ST 

2ND 

MIN 

SCORE 

40.  0* 

30.0* 

0.  0* 

0.  0* 

1 

3 

20.  0 * 

15.  0* 

2.  0* 

1.5* 

1 

3 

SPREAD* 

GOV’T  EST  COST 
COST  REALISM 

WEIGHTS 

TECHNICAL  <T> 

GOV’T  EST  COST  <E> 
COST  REALISM  <R> 
OTHER 
TOTAL 

WEIGHT  RATIOS 
E/T 
R/T 

E/(E+R> 

R/<E+R> 


A 

B 

60. 

0 

60. 

0 

3 

20. 

0 

15. 

0 

3 

10. 

0 

15. 

0 

3 

10. 

0 

10. 

0 

3 

100. 

0 

lOO. 

IT 

1 

or 

» 

33. 

3* 

25. 

0* 

3 

16. 

7* 

25. 

0* 

66. 

7* 

50. 

0* 

33. 

3* 

50. 

0* 

AFTER  ENTERING  OR 
CHANGING  DATA, 
PRESS  "F9"  TO 
RECALCULATE 


SPREAD  SENSITIVITY  (WITH  WEIGHT  ,*A,,)s 


OFFEROR 

- SCO 

RES - 

- GOV’ T 

EST - 

- REALISM- 

SPREAD 

1ST 

2ND 

1ST 

2ND 

A 

10.0 

10.0 

10.  0 

9.6 

B 

7.2 

6.3 

4.5 

2.4 

C 

9.  1 

8.  8 

8.0 

7.0 

D 

2.0 

1.0 

7.2 

5.9 

E 

1.0 

1.  0 

1.0 

1.0 

OFFEROR 

— ucrriauTcn  crn  dcc 

Wt  1  wn  1  C.U  oLUittp 

- GOV*  T 

PCT- — — 

T  CM. 

to  1 

VttHL  *  on 

- 1  U  1  HL“ 

BPREAD 

1ST 

2ND 

1ST 

2ND 

1ST 

2ND 

A 

20.0 

19.9 

10.0 

9.6 

30.0 

29.5 

B 

14.4 

12.  5 

4.5 

2.4 

18.9 

14.9 

C 

18.  1 

17.5 

8.  0 

7.0 

26.  1 

24.5 

D 

4.  1 

2.0 

7.2 

5.  9 

11.2 

7.9 

E 

2.0 

2.0 

1.0 

1.0 

3.0 

3.0 

WEIGHT  SENSITIVITY  (WITH  1ST  SPREAD) s 


- SC0RE8 - 

OFFEROR  GOV’T  EST  REALISM 


A 

B 

C 

D 

E 


10.0  10.0 

7.2  4.5 

9.  1  8. O 

2.  O  7.  2 

1.0  1.0 


A 


Table  2  (con't) 


OFFEROR 

- GOV*  T 

EST - 

— WEIGHTED  SCORES - 

- REALISM - 

- TOTAL- 

““  — 

WEIGHT 

A 

B 

A 

B 

A 

B 

A 

£0.0 

15.0 

10.  0 

14.9 

30.0 

£9.9 

B 

14.  4 

10.8 

4.  5 

6.6 

18.  9 

17.6 

C 

18.  1 

13.  6 

8.  0 

12.  0 

£6.  1 

£5.6 

D 

4.  1 

3.  0 

7.  £ 

10.  7 

11.  £ 

13.  8 

E 

£.0 

1.5 

1.  0 

1.5 

3.  0 

3.  0 

INVOKE  GRAPHS*  PRESS  "ALT"  KEY  AND  INDICATED  LETTER  SIMULTANEOUSLY 

-  (IF  THERE  IS  A  GRAPH  ON  THE  SCREEN,  FIRST  PRESS  ANY  KEY) 

(TO  RETURN  TO  THE  WORKSHEET,  PRESS  "Q") 

WEIGHTED 

SCORES  SCORES 
SPREAD  G  H 

WEIGHT  I 

PRINT  TABLE*  FOLLOW  LOTUS  INSTRUCTIONS 


PRINT  GRAPHS*  SAVE  EACH  GRAPH  TO  A  "PRINT-GRAPH"  FILE  AS  FOLLOWS* 


INVOKE  GRAPH  AS  ABOVE,  PRESS  ANY  KEY,  PRESS  "S",  AND 
TYPE  IN  A  "PRINT-GRAPH"  NAME  USING  APPROPRIATE  LETTER 
FROM  ABOVE  TABLE 

IF  MORE  THAN  ONE  GRAPH  IS  TO  BE  SAVED  OF  THE  SAME  TYPE 
(I.E. ,  DIFFERENT  ENTRIES  IN  THE  TABLE),  USE  A  DIFFERENT 
NAME  FOR  EACH  GRAPH — FOR  EXAMPLE,  Gl,  HI,  ETC. 

THEN,  USE  THE  LOTUS  "PRINT-GRAPH"  DISK 


Figure  7. 

SPREAD  SENSITIVITY 

SCORES-TRAPEZDID 


GOV'T  EST 
CT7T  1ST  SPREAD 


_  REALISM 

IV*3  2ND  SPREAD 


D  E 


Figure  8. 

SPREAD  SENSITIVITY 

V/EGHTED  9CORES-TTCAPEZOID 
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Figure  9. 


WEIGHT  SENSITIVITY 


WEIGHTED  SCOnra-TRAPEZOID 


U~7\  WEIGHT  "A" 


WEIGHT  "B" 


Table  3. 


COST  EVALUATION  SCORING  COMPARISON — TRAPEZOID  SHAPE  C6CCOMPT3 


ENTER i  SPREADS  AS  DECIMALS 


-  WEIGHTS  AS  WHOLE 

NUMBERS  -OR-  ENTER  WEIGHT 

RATIOS  AS 

DECIMALS 

SPREAD* 

CASE  1 

CASE  2 

CASE  3 

BASE-GOV’T  EST 

30.  0* 

40.  OX 

50.  OX 

3 

BASE-COST  REALISM 

15.  OX 

20.  OX 

25.  OX 

3 

TOP-GOV’T  EST 

0.  OX 

0.  OX 

0.  OX 

3 

TOP-COST  REALISM 

1.5X 

2.  OX 

2.  5X 

3 

MIN  SCORE— GV’T  EST 

1 

1 

1 

3 

MIN  SCORE-REALISM 

1 

1 

1 

3 

WEIGHT* 

TECHNICAL  <T> 

60.0 

60.0 

60.0 

3 

AFTER 

ENTERING  OR 

GOV’T  EST  COST  <E> 

20.0 

20.0 

20.0 

3 

CHANGING 

DATA, 

COST  REALISM 

<R> 

10.0 

10.0 

10.0 

3 

PRESS 

"F9"  TO 

OTHER 

10.0 

10.  0 

10.0 

3 

RECALCULATE 

TOTAL 

160. 0 

100.6 

100.  0 

• 

• 

— 

— 

or 

WEIGHT  RATIO* 

« 

a 

E/T 

33.  3X 

33.  3X 

33.  3X 

3 

R/T 

16.  7X 

16.  7X 

16.  7X 

3 

E/(E+R> 

66.  7X 

66.  7X 

66. 

R/(E+R> 

33.  3X 

33.  3X 

33.  3X 

OFFER  GOV' 

T  EST 

(G— G*)/G* 

(C-G) /6 

OFFEROR  <C>  (G) 

(DELTA) 

(DELTA’ ) 

A 

47.0 

48.0 

O.OX 

-2.  IX 

B 

47.0 

54.0 

12.  5X 

-13. OX 

C 

53.0 

50.0 

4.2X 

6.  OX 

D 

60.0 

65.0 

35.  4X 

-7.7X 

E 

89.0 

70.0 

45.  8X 

27.  IX 

i.ier  t  f2UTcn 

er 

norc 

CASE  1  *  - 

— -"SUUntb" 

’•wtion  i  C.U 

OFFEROR  GOV’T  EST  REALISM 

GOV’T  EST 

REALISM 

TOTAL 

X 

OF  MAX 

A 

10.0 

9.  6 

20.0 

9.6 

29.6 

99X 

B 

6.  3 

2.  4 

12.  5 

2.  4 

14.9 

50X 

C 

8.  a 

7.  0 

17.5 

7.  O 

24.  5 

82X 

D 

1.0 

5.9 

2.  0 

5.9 

7.9 

26X 

E 

1.0 

1.0 

2.0 

1.0 

3.0 

10X 

_ Lie:  T  QLlTCn 

er 

noce 

UHbt  cl  —  - 

KLw 

Wt  1  lari  i  C-D 

OFFEROR  GOV’ 

T  EST  REALISM 

GOV’T  EST 

REALISM 

TOTAL 

X 

OF  MAX 

A 

10.0 

10.0 

20.0 

10.0 

30.0 

100X 

B 

7.2 

4.5 

14.4 

4.5 

18.9 

63X 

C 

9.  1 

8.0 

18.  1 

8.0 

26.  1 

87X 

D 

2.  0 

7.2 

4.  1 

7.2 

11.2 

37X 

E 

1.0 

1.0 

2.0 

1.0 

LIC  T  QUTCA 

er 

3.0 

‘nocc _ 

10X 

CASE  3 1  - 

- - StUHtb- 

we X  Un 1 tu 

OFFEROR  GOV* 

T  EST  REALISM 

GOV’T  EST 

REALISM 

TOTAL 

X 

OF  MAX 

A 

10.0 

10.0 

20.0 

10.  0 

30.0 

100X 

B 

7.8 

5.8 

15.5 

5.8 

21.3 

71X 

C 

9.3 

8.6 

18.5 

8.6 

27.  1 

90X 

D 

3.  6 

7.9 

7.3 

7.9 

15.2 

51X 

E 

1.8 

1.0 

3.5 

1.0 

4.5 

15X 

Table  3  (con't) 


QUICK  ENTRIES l 


TO  LOAD  COMPLETE  EXAMPLE,  PRESS  "ALT"  KEY  AND  "X"  SIMULTANEOUSLY 
TO  RETURN  TO  BLANK  ENTRY  FORM,  PRESS  “ALT'*  KEY  AND  "Y"  SIMULTANEOUSLY 

INVOKE  ORAPHSi  PRESS  "ALT"  KEY  AND  INDICATED  LETTER  SIMULTANEOUSLY 

-  (IF  THERE  IS  A  GRAPH  ON  THE  SCREEN,  FIRST  PRESS  ANY  KEY) 

(TO  RETURN  TO  THE  WORKSHEET,  PRESS  "Q") 

SCORE  COMPARISON  A 

WEIGHTED  SCORE  COMPARISON  B 

PRINT  T ABLE I  PRESS  "ALT"  KEY  AND  "P"  SIMULTANEOUSLY 


PRINT  GRAPHS*  SAVE  EACH  GRAPH  TO  A  "PRINT-GRAPH"  FILE  AS  FOLLOWS* 


INVOKE  GRAPH  AS  ABOVE,  PRESS  ANY  KEY,  PRESS  "6",  AND 
TYPE  IN  A  "PRINT-GRAPH"  NAME  USING  APPROPRIATE  LETTER 
FROM  ABOVE  TABLE 

IF  MORE  THAN  ONE  GRAPH  IS  TO  BE  SAVED  OF  THE  SAME  TYPE 
(2.E. ,  DIFFERENT  ENTRIES  IN  THE  TABLE),  USE  A  DIFFERENT 
NAME  FOR  EACH  GRAPH — FOR  EXAMPLE,  Al,  Bl,  ETC. 

THEN,  USE  THE  LOTUS  "PRINT-GRAPH"  DISK 


_  GOVT  EST 

EZJ  CASE  1  P7\3  CASE  2 


ISM 

CASE  3 


Figure  11. 

SCORING  COMPARISON 


WEIGHTED  SCQWES-7RAPE2DID 


OPTEKOnS 
CASE  2 


1771  CASE  1 


ZF77X  case  3 


Table  4. 


COST  EVALUATION  SCORING  COMPARISON — TRAPEZOID  SHAPE  CSCCOMPT3 


ENTER i  SPREADS  AS  DECIMALS 

_  WEIGHTS  AS  WHOLE  NUMBERS  -OR-  ENTER  WEIGHT  RATIOS  AS  DECIMALS 


SPREAD i 

CASE  1 

CASE  2 

CASE  3 

BASE-GOV*  T 

EST 

40.  OX 

40.  OX 

40.  OX 

3 

BASE-COST 

REALISM 

£0.  OX 

20.  OX 

20.  OX 

3 

TOP-GOV*  T 

EST 

0.  OX 

0.  OX 

O.OX 

3 

TOP-COST  REALISM 

2.  OX 

2.  OX 

2.  OX 

3 

MIN  SCORE- 

•GV’T  EST 

1 

1 

1 

3 

MIN  SCORE- 

-REALISM 

1 

1 

1 

3 

WEIGHT i 

TECHNICAL 

(T) 

60.0 

60.0 

60.0 

3 

AFTER  1 

ENTERING  OR 

GOV’T  EST 

COST  (E) 

15.0 

20.0 

22.5 

3 

CHANGING  DATA, 

COST  REALISM 

<R> 

15.0 

10.0 

7.5 

3 

PRESS 

"F9"  TO 

OTHER 

10.0 

10.0 

10.0 

X 

RECALCULATE 

TOTAL 

100.  0 

100.0 

100.6 

! 

or 

WEIGHT  RATIO i 

1 

E/T 

25.  OX 

33.  3X 

37.5X1 

3 

R/T 

25.  OX 

16.  7X 

12.  5X 

3 

E/(E+R> 

50.  OX 

66.  7X 

75.  OX 

R/(E+R) 

50.  OX 

33.  3X 

25.  OX 

OFFER  GOV' 

T  EST 

<G— G*> /G* 

<C-G>/G 

OFFEROR 

<C>  <G> 

< DELTA > 

(DELTA*  > 

A 

47.0 

48.0 

o.  OX 

-2.  IX 

B 

47.0 

54.0 

12.  5X 

-13.  OX 

C 

53.0 

50.0 

4.2X 

6.  OX 

D 

60.0 

65.0 

35.  4X 

-7.7X 

E 

69.0 

70.0 

45.  ex 

27.  IX 

_  nr:  t  ouTCn 

cr 

knDce... 

CASE  li  - 

—SCORES- 

.  ■■■  * 

OFFEROR  GOV’T  EST  REALISM 

GOV’T  EST 

REALISM 

TOTAL 

X  OF 

MAX 

A 

10.0 

10.  0 

15.0 

14.9 

29.9 

1O0X 

B 

7.2 

4.5 

10.8 

6.8 

17.6 

59X 

C 

S.  1 

6.  0 

13.6 

12.0 

25.6 

85X 

D 

2.0 

7.2 

3.0 

10.  7 

13.8 

46X 

E 

1.0 

1.0 

1.5 

1.5 

LlCTfiUTCn 

i  cr 

3.0 

10X 

CASE  2 i  - 

— — SCORES 

WC.  A  on  1  L.W 

OFFEROR  GOV’T  EST  REALISM 

GOV’T  EST 

REALISM 

TOTAL 

X  OF 

MAX 

A 

10.0 

10.0 

20.0 

10.0 

30.0 

10OX 

B 

7.2 

4.5 

14.  4 

4.5 

18.9 

63X 

C 

9.  1 

8.0 

18.  1 

8.0 

26.  1 

87X 

D 

2.0 

7.2 

4.  1 

7.2 

11.2 

37X 

E 

1.0 

1.0 

2.0 

1.0 

1-lET  T  OUTCr 

k  or 

3.0 

*nocc_.. 

lOX 

CASE  3 i 

- SCORES 

OFFEROR  GOV’T  EST  REALISM 

GOV’T  EST 

REALISM 

TOTAL 

X  OF 

MAX 

A 

10.0 

10.0 

22.5 

7.5 

30.0 

1O0X 

B 

7.2 

4.5 

16.2 

3.4 

19.6 

65X 

C 

9.  1 

8.0 

20.4 

6.0 

26.4 

88X 

D 

2.0 

7.2 

4.6 

5.4 

9.9 

33X 

E 

1.0 

1.0 

2.3 

0.8 

3.0 

10X 

QUICK  ENTRIES* 


TO  LOAD  COMPLETE  EXAMPLE,  PRESS  "ALT"  KEY  AND  "X"  SIMULTANEOUSLY 
TO  RETURN  TO  BLANK  ENTRY  FORM,  PRESS  "ALT"  KEY  AND  "Y"  SIMULTANEOUSLY 

INVOKE  GRAPHS*  PRESS  "ALT"  KEY  AND  INDICATED  LETTER  SIMULTANEOUSLY 

-  (IP  THERE  IS  A  GRAPH  ON  THE  SCREEN,  FIRST  PRESS  ANY  KEY) 

<TO  RETURN  TO  THE  WORKSHEET,  PRESS  "Q") 

SCORE  COMPARISON  A 

WEIGHTED  SCORE  COMPARISON  B 

PRINT  TABLE*  PRESS  "ALT"  KEY  AND  "P"  SIMULTANEOUSLY 


PRINT  GRAPHS*  SAVE  EACH  GRAPH  TO  A  “PRINT-GRAPH"  FILE  AS  FOLLOWS* 


INVOKE  GRAPH  AS  ABOVE,  PRESS  ANY  KEY,  PRESS  "S",  AND 
TYPE  IN  A  "PRINT-GRAPH"  NAME  USING  APPROPRIATE  LETTER 
FROM  ABOVE  TABLE 

IF  MORE  THAN  ONE  GRAPH  IS  TO  BE  SAVED  OF  THE  SAME  TYPE 
(I.E. ,  DIFFERENT  ENTRIES  IN  THE  TABLE),  USE  A  DIFFERENT 
NAME  FOR  EACH  GRAPH — FOR  EXAMPLE,  Al,  Bl,  ETC. 

THEN,  USE  THE  LOTUS  "PRINT-GRAPH"  DISK 
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Figure  13. 

SCORING  COMPARISON 


NW3QHTED  SCQWES-TWAPEZQID 


1771  CASE  1 


V7Zk  CASE  3 


Table  5. 


COST  EVALUATION  TRADEOFF  MODEL — TRAPEZOID  SHAPE 


tTRDTRAPD 


ENTER i  SPREADS  AND  ENVELOPES  AS  DECIMALS 

-  WEIGHTS  AS  WHOLE  NUMBERS  -OR-  ENTER  WEIGHT  RATIOS  AS  DECIMALS 

APPROXIMATE  COST  IN  MILLIONS  OF  DOLLARS 


SPREAD i 

GOV’T  EST 
REALISM 


1ST 

- BASE — 

2ND 

3RD 

1ST 

-TOP - 

2ND 

3RD 

MIN 

SCORE 

30.0* 

40.  0* 

50.  0* 

0.  0* 

0.  0* 

O.  0* 

1 

3 

15.  OX 

20.  0* 

25.0* 

1.5* 

2.  0* 

2.5* 

1 

3 

WEIGHT i 


B 


TECHNICAL  <T) 

60.  0 

60.0 

60.  0 

GOV’T  EST  (E) 

15.0 

20.  0 

22.5 

COST  REALISM  <R> 

15.  0 

10.  0 

7.5 

OTHER 

10.0 

10.  0 

10.  0 

TOTAL 

100.0 

100.  0 

100.  0 

WEIGHT  RATIO: 
E/T 
R/T 

E/(E+R) 

R/ (E+R) 


B 


50.0* 

50.0* 


SS.  7* 
33.  3* 


3 

3 

3 

3 

I 

I 

or 


25.0* 

33.  3* 

37.  5* 

25.  0* 

16.7* 

12.5* 

75.0* 

25.0* 


APPROXIMATE  COST:  [ »50 |MI?  LION  (OPTIONAL)  3 


ENVELOPES:  TECHNICAL 
ADVANTAGE 


COST  WILLING  UNREALISM  WILLING 
TO  ACCEPT  TO  ACCEPT 

MIN  MAX  MIN  MAX 


5.0* 

3.8* 

8.  8* 

5.  0* 

12.5* 

10.0* 

7.  5* 

17.  5* 

7.5* 

17.  5* 

20.  0* 

15.  0* 

35.0* 

12.  5* 

25.0* 

5.0* 
10.  O* 
20.  O* 


3 
3 
3 
or 


*1.9 

*4.  4 

*2.5 

*6.  3 

3 

*3.  8 

*8.  8 

*3.  8 

*8.  6 

3 

*7.  5 

*17.  5 

*6.  3 

*12.  5 

3 

AFTER  ENTERING  OR  CHANGING  DATA,  PRESS  "F9"  TO  RECALCULATE 


QUICK  ENTRIES: 


TO  LOAD  ILLUSTRATIVE  ENVELOPE,  PRESS  •'ALT”  KEY  AND  "VM  SIMULTANEOUSLY 
TO  LOAD  COMPLETE  EXAMPLE,  PRESS  "ALT"  KEY  AND  "XM  SIMULTANEOUSLY 
TO  RETURN  TO  BLANK  ENTRY  FORM,  PRESS  "ALT”  KEY  AND  "Y"  SIMULTANEOUSLY 


INVOKE  GRAPHS:  PRESS  "ALT"  KEY  AND  INDICATED  LETTER  SIMULTANEOUSLY 

-  <IP  THERE  IS  A  GRAPH  ON  THE  SCREEN,  FIRST  PRESS  ANY  KEY) 

(TO  RETURN  TO  THE  WORKSHEET,  PRESS  "Q") 

1ST  2ND  3RD  SPREAD  BAND 

SPREAD  SPREAD  SPREAD  DIFFERENCE 
GOV’T  EST  COST  A  D  G  J  M 

COST  REALISM  B  E  H  K  N 

GOV’T  EST/REALISM  C  F  I  L 


NOTE:  SPREAD  DIFFERENCE  FOR  WT  "B".  BAND  FOR  2ND  SPREAD  A  WT  "B" 


Table  5  (con't) 


PRINT  TABLE »  PRESS  "ALT"  KEY  AND  "P"  SIMULTANEOUSLY 


PRINT  GRAPHS*  SAVE  EACH  GRAPH  TO  A  "PRINT-GRAPH"  FILE  AS  FOLLOWS* 


INVOKE  GRAPH  AS  ABOVE,  PRESS  ANY  KEY,  PRESS  "S",  AND 
TYPE  IN  A  "PRINT-GRAPH"  NAME  USING  APPROPRIATE  LETTER 
FROM  ABOVE  TABLE 

IF  MORE  THAN  ONE  GRAPH  IS  TO  BE  SAVED  OF  THE  SAME  TYPE 
<I.E. ,  DIFFERENT  ENTRIES  IN  THE  TABLE),  USE  A  DIFFERENT 
NAME  FOR  EACH  GRAPH — FOR  EXAMPLE,  Al,  Bl,  ETC. 

THEN,  USE  THE  LOTUS  "PRINT-GRAPH"  DISK 


COST  REALISM  DELTA  FOR  OFFEOR  A  COST  REALISM  DELTA  FOR 


Figure  14. 

GOVT  EST/REALISM  TRADEOFF  GRAPH 


SECOND  SPrCAD-TfTAPCZOID 


ESfT.  COST  ADVANTAGE  OF  A  ILLATIVE  TO  B 
O  WEIGHT  "A"  O  WEIGHT  “C"  A  WEIGHT  "B" 

Figure  15. 

GOVT  EST/REALISM  TRADEOFF  GRAPH 


SPFEAO  OirrEfENCE  FDR  WEJGHT  “S" 


ESfT.  COST  ADVANTAGE  OF  A  RELATTVE  TO  B 

C  9ID  SPRSAD  A  2ND  SPFEAD 


1ST  SPREAD 


OOST  REALISM  DELTA  FOR  OFFEOR  A  EST.  COST  DISADVANTAGE  OF  A  REL.  TO  B 


Figure  16 


GOVT  ESTIMATED  COST  TRADEOFF  GRAPH 


SECOND  SPflEAD-TRAPEZOID 


TECHNICAL  ADVANTAGE  OF  A  RELATIVE  TO  B 
O  WEIGHT  "A"  A  WEIGHT  "CM  A  WEIGHT  "B" 


Figure  17. 

COST  REALISM  TRADEOFF  GRAPH 


SECOND  SPREAD-TRAPEZOID 


TECHNICAL  ADVANTAGE  OF  A  RELATIVE  TO  B 
O  WEIGHT  "A"  C  WEIGHT  "C"  A  WEIGHT  HB" 


Figure  18. 

GOV'T  ESTIMATED  COST  TRADEOFF  GRAPH 

SPWLAD  DIFFERENCE  FOR  WEIGHT  "B" 


BOOK 


40. OX 


30. OX 


TECHNICAL  ADVANT AGE  OF  A  RELATIVE  TO  B 
1ST  SPREAD  O  3RD  SPREAD  A  2ND  SPPEAD 


Figure  19. 

COST  REALISM  TRADEOFF  GRAPH 

SPREAD  DIFFEfCNCE  FOR  WEIGHT  "H" 


COST  REALISM  DELTA  FOR  OFFEC-R  A  EST.  COST  DISADVANTAGE  OF  A  REL.  TO  B 


Figure  20. 

GOV'T  ESTIMATED  COST  TRADEOFF  GRAPH 

BAND -SECOND  SPREAD  Ac  WEIGHT  "B" 


TECHNICAL  ADVANTAGE  OF  A  RELATIVE  TO  B 
3ASEUNE  «  Bs  TECH  70%  OF  MAX  &  Bs  DELTA'  SSC 


Figure  21. 


COST  REALISM  TRADEOFF  GRAPH 

BAND-SECOND  SPREAD  <fc  WEIGHT  "H" 


TECHNICAL  ADVANTAGE  OF  A  RELATIVE  TO  B 
C  Bs  TECH  70*  OF  MAX  A  Bs  DELTA  S* 


1  BASELINE 


COST  REALISM  DELTA  FOR  OFFEOR  A  COST  REALISM  DELTA  FOR 


< 


Figure  22. 

GOVT  EST/REALISM  TRADEOFF  GRAPH 


flBST  SPREAD-TRAPEZOID 


D  WEIGHT  "A" 


csnr.  cosnr  advantage  of  a  helative  to  s 

O  WEIGHT  "C"  A  WEIGHT  "B" 


Figure  23. 

GOVT  EST/REALISM  TRADEOFF  GRAPH 


THIfO  SPREAD-TRAPEZOID 


EST.  COST  ADVANTAGE  OF  A  RELATIVE  TO  B 

A  WRinHT  •*r«  A  WEIGHT 


O  WEIGHT  "A" 


Figure  24. 

GOV'T  ESTIMATED  COST  TRADEOFF  GRAPH 

FIRST  SPREAD— TRAPEZOID 


33.3% 


TECHNICAL  ADVANTAGE  OF  A  RELATIVE  TO  B 
WEIGHT  "A"  *  WEIGHT  “C"  A  WEIGHT  "B" 


Figure  25. 

COST  REALISM  TRADEOFF  GRAPH 

FIRST  SPREAD-TRAPEZOID 


2S.OX 


TECHNICAL  ADVANTAGE  OF  A  RELATIVE  TO  B 
WEIGHT  “A"  C  WTIGHT  "C"  A  WEIGHT  "i" 


Figure  26. 

GOV'T  ESTIMATED  COST  TRADEOFF  GRAPH 

THITO  SPREAD-TRAPEZOID 


2S.OJC 


33.3* 


37.fi* 


TECHNICAL  A  WANTAGE  OF  A  ft  ELAT!  VC  TO  B 
WEIGHT  “A"  O  WEIGHT  “C"  A  WEIGHT  ,,B" 


Figure  27. 

COST  REALISM  TRADEOFF  GRAPH 

THIRD  9PFEAD-TRAPEZOID 


TECHNICAL  ADVANTAGE  OF  A  RELATIVE  TO  B 
WEIGHT  "A"  A  WEIGHT  "C"  A  WEIGHT  "B" 


Table  6. 


COST  EVALUATION  TRADEOFF  MODEL — TRAPEZOID  SHAPE  CTRDTRAP3 


ENTER i  SPREADS  AND  ENVELOPES  AS  DECIMALS 

-  WEIGHTS  AS  WHOLE  NUMBERS  -OR-  ENTER  WEIGHT  RATIOS  AS  DECIMALS 

APPROXIMATE  COST  IN  MILLIONS  OF  DOLLARS 


SPREAD: 

r  1ST 

— BASE- 
2ND 

3RD 

1ST 

-TOP - 

2ND 

_  3RD 

MIN 

SCOfi 

1 

E 

GOV’T  EST 

30.  0* 

40.  OX 

50.  OX 

0.  OX 

0.  OX 

0.  OX 

1 

3 

REALISM 

15.  OX 

20.  OX 

25.  OX 

1. 5X 

2.  OX 

2.  5X 

1 

3 

WEIGHT: 

A 

B 

_C 

TECHNICAL  (T) 

70.  0 

60.  0 

55.  0 

3 

GOV’T  EST  (E) 

13.  3 

20.  0 

23.  3 

3 

COST  REALISM  (R) 

6.  7 

10.  0 

11.7 

3 

OTHER 

1 

10.0 

10.  0 

10.  0 

3 

TOTAL 

100.0 

100.  0 

~106.  0 

1 

1 

or 

WEIGHT  RATIO: 

A 

B 

_ c 

1 

1 

E/T 

19.  OX 

33.  3X 

42.  4X 

3 

R/T 

9.5X 

16.  7X 

21. 2X 

3 

E/(E+R> 

66. 7x 

66.  7X 

66.7* 

R/ (E+R) 

33.  3X 

33.  3X 

33.  3X 

APPROXIMATE  COST: 


»50|MILLI0N  (OPTIONAL)  3 


ENVELOPES :  TECHNICAL  COST  WILLING  UNREALISM  WILLING 


ADVANTAGE 

TO 

ACCEPT 

TO 

ACCEPT 

MIN 

MAX 

MIN 

MAX 

5.  OX 

3.  8X 

6.  8X 

5.  OX 

12.  5X 

10.  OX 

7.  5X 

17.  5 X 

7.  5X 

17.  5X 

20.  OX 

15.  OX 

35.  OX 

12.  5X 

25.  OX 

5.  OX 

*1.9 

*4.  4 

*2.5 

*6.  3 

10.  OX 

*3.  8 

*8.  8 

*3.  8 

*8.8 

20.  OX 

*7.  5 

*17.  5 

*6.  3 

*12.  5 

3 

13 

3 

or 

3 

3 

3 


AFTER  ENTERING  OR  CHANGING  DATA,  PRESS  "F9"  TO  RECALCULATE 
QUICK  ENTRIES: 


TO  LOAD  ILLUSTRATIVE  ENVELOPE,  PRESS  "ALT"  KEY  AND  "V"  SIMULTANEOUSLY 
TO  LOAD  COMPLETE  EXAMPLE,  PRESS  "ALT"  KEY  AND  "X"  SIMULTANEOUSLY 
TO  RETURN  TO  BLANK  ENTRY  FORM,  PRESS  "ALT"  KEY  AND  "Y"  SIMULTANEOUSLY 

INVOKE  GRAPHS:  PRESS  "ALT"  KEY  AND  INDICATED  LETTER  SIMULTANEOUSLY 
-  (IF  THERE  IS  A  GRAPH  ON  THE  SCREEN,  FIRST  PRESS  ANY  KEY) 


(TO 

RETURN  TO  THE 

WORKSHEET 

,  PRESS 

"Q") 

1ST 

2ND 

3RD 

SPREAD 

BAND 

SPREAD 

SPREAD 

SPREAD 

DIFFERENCE 

GOV’T  EST  COST 

A 

D 

G 

J 

M 

COST  REALISM 

B 

E 

H 

K 

N 

SOV’T  EST/ REAL ISM 

C 

F 

I 

L 

NOTE:  SPREAD  DIFFERENCE  FOR  WT  "B" 


BAND  FOR  2ND  SPREAD  ft  WT  "B" 


Table  6  (con't) 


PRINT  TABLE:  PRESS  "ALT"  KEY  AND  "P"  SIMULTANEOUSLY 


PRINT  GRAPHS:  SAVE  EACH  GRAPH  TO  A  *' PRINT-GRAPH"  FILE  AS  FOLLOWS: 


INVOKE  GRAPH  AS  ABOVE,  PRESS  ANY  KEY,  PRESS  "S",  AND 
TYPE  IN  A  "PRINT-GRAPH"  NAME  USING  APPROPRIATE  LETTER 
FROM  ABOVE  TABLE 

IF  MORE  THAN  ONE  GRAPH  IS  TO  BE  SAVED  OF  THE  SAME  TYPE 
<I.E. ,  DIFFERENT  ENTRIES  IN  THE  TABLE),  USE  A  DIFFERENT 
NAME  FOR  EACH  GRAPH — FOR  EXAMPLE,  AI,  Bl,  ETC. 

THEN,  USE  THE  LOTUS  "PRINT-GRAPH"  DISK 


Figure  28. 

GOV'T  ESTIMATED  COST  TRADEOFF  GRAPH 


riRST  SPREAD-TRAPEZOID 


1»OX 


42.4X 


TECHNICAL.  ADVANTAGE  OR  A  RELATIVE  TO  B 
WEIGHT  "A"  O  WEIGHT  ,,CM  A  WEIGHT  "B" 


Figure  29. 

COST  REALISM  TRADEOFF  GRAPH 

PIRST  SPREAD-TRAPEZOID 


TECHNICAL  ADVANTAGE  OR  A  RELATIVE  TO  B 
WEIGHT  "A"  A  WEIGHT  "C"  A  WEIGHT  "B" 


Figure  30. 

GOV'T  ESTIMATED  COST  TRADEOFF  GRAPH 


SECOND  SPrtEAD-TBAPEZQID 


1SKOX 


33  3X 


42.4X 


TECHNICAL  ADVANTAGE  OF  A  HELA71VE  TO  B 
WEIGHT  "A"  O  WEIGHT  "C"  A  WEIGHT  •’B" 


Figure  31. 

COST  REALISM  TRADEOFF  GRAPH 

SECOND  SPBEAD-TKAPEZOID 


g  1A% 


TECHNICAL  ADVANTAGE  OF  A  RELATIVE  TO  B 
WEIGHT  “A"  O  WEIGHT  "C"  A  WEIGHT  MB 


Figure  32. 

GOV'T  ESTIMATED  COST  TRADEOFF  GRAPH 

THI m  SPfCAD-TRArezOID 


33.3K 


43.4X 


-TECHNICAL  ADVANTAGE  OF  A  RELATIVE  TO  B 
WEIGHT  "A"  A  WEIGHT  “C"  A  WEIGHT  MB" 


Figure  33. 

COST  REALISM  TRADEOFF  GRAPH 

THIRD  SPRSAD-TRAPEZQID 


TECHNICAL  ADVANTAGE  OF  A  RELATIVE  TO  B 
WEIGHT  "A"  A  WEIGHT  "CM  A  WEIGHT  "fl 


Figure  34. 


SCORING  RULE  COMPARISON 


GOV'T  ESTIMATE 


DELTA 


Figure  35. 

SCORING  RULE  COMPARISON 


COST  fCALISM 


*  » . ■ .  I .  » . I . . ■ . « .  I - - - "I  "  *  —  "  l  "  —  ’  —  --  — 

aox-aax -aox- lax- tox  -ax  ox  ax  iox  isx  aox  aax  aox 

delta' 


Table  7. 


COST  EVALUATION  SCORING  MODEL — NORMAL  CURVE  SHAPE  CSCNORM3 


ENTER*  SPREADS  AS  DECIMALS 

-  WEIGHTS  AS  WHOLE  NUMBERS  -OR-  ENTER  WEIGHT  RATIOS  AS  DECIMALS 


SPREAD! 


GOV’T  EST 

COST 

40.0* 

COST  REALISM 

1 

20.0* 

WEIGHT! 

TECHNICAL 

<T> 

60.0 

3 

GOV’T  EST 

COST  (E) 

20.  0 

3 

AFTER 

ENTERING 

OR 

CHANGING 

DATA, 

COST  REALISM 

(R) 

10.0 

3 

PRESS 

"F9"  TO  RECALCULATE 

OTHER 

10.  0 

3 

— 

TOTAL 

100.0 

• 

1 

or 

WEIGHT  RATIO! 

,1 

E/T 

33.  3* 

R/T 

16.  7* 

J3 

E/(E+R> 

66.  7* 

R/(E+R> 

33.  3* 

OFFER  GOV* 

T  EST 

(G 

-G*>/G* 

<C-G>/6  - 

- — - 

- SCORES - 

OFFEROR 

(C)  (G) 

(DELTA! 

(DELTA’  )  GOV’ 

T  EST  1 

REALISM 

A 

47.0 

46.0 

0.0* 

-2.  1* 

10.0 

9.8 

B 

47.0 

54.0 

12.  5* 

-13.  0* 

7.  1 

4.3 

C 

53.  0 

50.0 

4.2* 

6.  O* 

9.  1 

8.4 

D 

60.0 

65.0 

35.  4* 

-7.7* 

2.  1 

7.4 

E 

69.0 

70.0 

45.6* 

27.  1* 

1.0 

0.3 

rrcuTcn 

Qi 

rnocrc^ 

TECH  ADV 

TRADEOFF 

1 1  lari  1  C.U 

OFFEROR  GOV’T  EST  REALISM 

TOTAL 

*  OF  MAX 

LOW 

HIGH 

A 

20.  0 

9.8 

29.8 

99* 

0* 

0* 

B 

14.  2 

4.3 

18.5 

62* 

23* 

27* 

C 

16.  3 

6.4 

26.6 

89* 

6* 

8* 

D 

4.3 

7.  4 

11.7 

39* 

43* 

43* 

E 

1.9 

0.  3 

2.2 

7* 

85* 

85* 

nCI  Tfi  DQMOC 

i 

\ 

— ■ 

— GOV’T  EST  COST — 

~ 

— COST  REALISM - 

OFFEROR 

LOW 

HIGH 

LOW 

HIGH 

A 

0.  0* 

2.0* 

-0.  1* 

-4.  1* 

B 

10.5* 

14.  5* 

-11.0* 

-15.0* 

C 

2.2* 

6.  2* 

4.  0* 

8.0* 

D 

33.  4* 

37.4* 

-5.7* 

-9.7* 

E 

43.  6* 

47.  8* 

25.  1* 

29.  1* 

ernoc 

D 

OMQCT.  _ 

lie*  t  ouTcn  ernoc  domrc 

"“□Ll/nt 

ft 

HIMUt 

- 

— GOV’T  EST  COST — 

- 

— COST  REALISM - 

TOTAL 

TOTAL 

OFFEROR 

LOW 

HIGH 

LOW 

HIGH 

LOW 

HIGH 

A 

9.6 

10.0 

9.2 

10.0 

28.  4 

30.0 

B 

6.6 

7.6 

3.3 

5.  5 

16.4 

20.7 

C 

8.  7 

9.6 

7.  3 

9.2 

24.6 

28.4 

D 

1.9 

2.  4 

6.  3 

8.5 

10.  0 

13.  4 

E 

0.6 

1.  1 

0.  1 

0.4 

1.6 

2.7 

Table  7  (con't) 

QUICK  ENTRIES! 


TO  LOAD  COMPLETE  EXAMPLE,  PRESS  "ALT'*  KEY  AND  "X"  SIMULTANEOUSLY 
TO  RETURN  TO  BLANK  ENTRY  FORM,  PRESS  "ALT"  KEY  AND  "YH  SIMULTANEOUSLY 

INVOKE  GRAPHS!  PRESS  "ALT"  KEY  AND  INDICATED  LETTER  SIMULTANEOUSLY 

-  < IF  THERE  IS  A  GRAPH  ON  THE  SCREEN,  FIRST  PRESS  ANY  KEY) 

(TO  RETURN  TO  THE  WORKSHEET,  PRESS  MG"> 

GOV’T  EST  REALISM  WEIGHTED 
SCORE  ABC 

RANGE  D  E  F 

PRINT  TABLE!  PRESS  "ALT"  KEY  AND  "P"  SIMULTANEOUSLY 


PRINT  GRAPHS:  SAVE  EACH  GRAPH  TO  A  "PRINT -GRAPH"  FILE  AS  FOLLOWS! 


INVOKE  GRAPH  AS  ABOVE,  PRESS  ANY  KEY,  PRE8S  "8",  AND 
TYPE  IN  A  "PRINT-GRAPH"  NAME  USING  APPROPRIATE  LETTER 
FROM  ABOVE  TABLE 

IF  MORE  THAN  ONE  GRAPH  IS  TO  BE  SAVED  OF  THE  SAME  TYPE 
(I.E. ,  DIFFERENT  ENTRIES  IN  THE  TABLE),  USE  A  DIFFERENT 
NAME  FOR  EACH  GRAPH — FOR  EXAMPLE,  Al,  Bl,  ETC. 

THEN,  USE  THE  LOTUS  "PRINT-GRAPH"  DISK 

SENSITIVITY  ANALYSIS!  GO  TO  NEXT  PART 


Figure  36. 

ESTIMATED  COST  SCORE 


Ul 


Figure  37. 

COST  REALISM  SCORE 


(C-OVO  (OQ_TA') 


Figure  38. 

TRAPEZOID  AND  NORMAL  SCORES  COMPARED 


ABODE  ABODE 

GOVT  ESTIMATE  COST  REALISM 


Figure  39. 

TRAPEZOID  AND  NORMAL  SCORES  COMPARED 


ABODE 


OFJEBOR 


Table  8. 


COST  EVALUATION  TRADEOFF  MODEL — NORMAL  CURVE  SHAPE  CTRDNORM3 


ENTER i  SPREADS  AND  ENVELOPES  AS  DECIMALS 

-  WEIGHTS  AS  WHOLE  NUMBERS  -OR-  ENTER  WEIGHT  RATIOS  AS  DECIMALS 

APPROXIMATE  COST  IN  MILLIONS  OF  DOLLARS 


SPREADi 

GOV’T  EST  COST 
COST  REALISM 

WEIGHT! 

TECHNICAL  (T) 
GOV’T  EST  (E) 
COST  REALISM  <R> 
OTHER 
TOTAL 

WEIGHT  RATIO; 

E/T 

R/T 

E/ (E+R) 

R/ (E+R) 


1ST 

2ND 

3RD 

30.0* 

40.  0* 

50.  0* 

3 

15.  0* 

20.  0* 

25.  0* 

3 

A 

B 

C 

60.0 

60.  0 

60.  0 

3 

15.0 

20.  0 

22.  5 

3 

15.  0 

10.  0 

7.  5 

3 

10.  0 

10.  0 

10.  0 

3 

100.  0 

100.  0 

100.  0 

1 

1 

or 

A 

B 

C 

1 

1 

25.  0* 

33.  3* 

37.  5* 

3 

25.  0* 

16.  7* 

12.  5* 

3 

50.0* 

66.  7* 

75.  0* 

50.  0* 

33.  3* 

25.  0* 

APPROXIMATE  COST t 


|  *50 | MILL I ON  (OPTIONAL)  3 


ENVELOPES;  TECHNICAL  COST  WILLING 

ADVANTAGE  TO  ACCEPT 

MIN  MAX 


UNREALISM  WILLING 
TO  ACCEPT 
MIN  MAX 


5.  O* 

3.  8* 

8.  8* 

5.  0* 

12.  5* 

10.  O* 

7.  5* 

17.  5* 

7.  5* 

17.  5* 

20.  O* 

15.  O* 

35.  0* 

12.  5* 

25.  O* 

or 


5.  O* 

*1.9 

*4.  4 

*2.  5 

*6.  3 

10.  O* 

*3.  8 

*8.  8 

*3.  8 

*8.  8 

20.  O* 

*7.  5 

*17.  5 

*6.  3 

*12.  5 

AFTER  ENTERING  OR  CHANGING  DATA,  PRESS  "F9"  TO  RECALCULATE 


QUICK  ENTRIES; 


TO  LOAD  ILLUSTRATIVE  ENVELOPE,  PRESS  "ALT"  KEY  AND  "V"  SIMULTANEOUSLY 
TO  LOAD  COMPLETE  EXAMPLE,  PRESS  "ALT"  KEY  AND  "X"  SIMULTANEOUSLY 
TO  RETURN  TO  BLANK  ENTRY  FORM,  PRESS  "ALT"  KEY  AND  "Y"  SIMULTANEOUSLY 

INVOKE  GRAPHS;  PRESS  "ALT"  KEY  AND  INDICATED  LETTER  SIMULTANEOUSLY 
-  < IP  THERE  IS  A  GRAPH  ON  THE  SCREEN,  FIRST  PRESS  ANY  KEY) 


(TO  RETURN 

TO  THE 

WORKSHEET, 

PRESS  "Q 

"> 

1ST 

2ND 

3RD 

SPREAD 

BAND 

SPREAD 

i  SPREAD 

SPREAD 

DIFFERENCE 

GOV’T  EST  COST 

A 

D 

G 

J 

M 

COST  REALISM 

B 

E 

H 

K 

N 

GOV’T  EST/ REAL ISM 

C 

F 

I 

L 

NOTE!  SPREAD  DIFFERENCE 

FOR  WT 

"B".  BAND 

FOR  2ND 

SPREAD  ft  WT 

“B" 

Table  8  (con't) 

PRINT  TABLE*  PRESS  "ALT"  KEY  AND  "P"  SIMULTANEOUSLY 


PRINT  GRAPHS*  SAVE  EACH  GRAPH  TO  A  “PRINT-GRAPH"  FILE  AS  FOLLOWS* 


INVOKE  GRAPH  AS  ABOVE,  PRESS  ANY  KEY,  PRESS  "S",  AND 
TYPE  IN  A  "PRINT-GRAPH"  NAME  USING  APPROPRIATE  LETTER 
FROM  ABOVE  TABLE 

IF  MORE  THAN  ONE  GRAPH  IS  TO  BE  SAVED  OF  THE  SAME  TYPE 
(I.E. ,  DIFFERENT  ENTRIES  IN  THE  TABLE),  USE  A  DIFFERENT 
NAME  FOR  EACH  GRAPH — FOR  EXAMPLE,  Al,  Bl,  ETC. 

THEN,  USE  THE  LOTUS  "PRINT-GRAPH"  DISK 


COST  REALISM  DELTA  FOR  OFFECR  A  COST  REALISM  DELTA  FOR  C-FFEOR  A 


Figure  40. 


GOVT  EST/REALISM  TRADEOFF  GRAPH 


SECOND  SPREAD- NORM AL 


csrr.  cost  advantage  of  a  relative  to  b 

WEIGHT  ••A*1  A  WEIGHT  ”C"  A  WEIGHT  ,•B,, 


Figure  41. 

GOVT  EST/REALISM  TRADEOFF  GRAPH 

SPREAD  DlfTCfCNCE  FOR  WEJGHT  "B" 


ESI.  COST  ADVANTAGE  OF  A  FtLATTVE  TO  B 
1ST  SPfCAD  O  JtD  SPREAD  A  2ND  SPREAD 


COST  REALISM  DELTA  FOR  OFFEOR  A  EST.  COST  DISADVANTAGE  OF  A  REL.  TO  B 


Figure  42. 

GOV'T  ESTIMATED  COST  TRADEOFF  GRAPH 

SECOND  SPREAD-NORMAL. 


A3  3X 
37.6JC 


O  WEIGHT  ”A" 


TECHNICAL  ADVANTAGE  Or  A  RELATIVE  TO  B 

O  WEIGHT  "C"  A  WEIGHT  "B" 


Figure  43. 
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Table  9 

SCTRAP  Model  Listing 


Spread  and  Minimum  Score  Inputs 


Gov't  Est  Base 

C7  : 

(PI) 

U 

0 

Gov't  Est  Top 

D7 : 

(PI) 

U 

0 

Gov't  Est  Min  Score 

E7 : 

(F0) 

u 

0 

Cost  Realism  Base 

C8 : 

(PI) 

u 

0 

Cost  Realism  Top 

D8 : 

(PI) 

u 

0 

Cost  Realism  Min  Score 

E8 : 

(F0) 

u 

0 

Weight  Inputs 


Technical 

Cll : 

(FI) 

U 

(C15-C14)/( 1+C1 8+C1 9 ) 

Gov't  Est  Cost 

Cl  2 : 

(FI) 

u 

+C1 1*C18 

Cost  Realism 

C13 : 

(FI) 

u 

+C1 1*C19 

Other 

C14 : 

(FI) 

u 

0 

Total 

Cl  5 : 

(FI) 

u 

100 

Weight  Ratios 

E/T 

C18 : 

(PI) 

u 

+C12/C1 1 

R/T 

C19: 

(PI) 

u 

+C13/C1 1 

E/(E+R) 

C20 : 

(PI) 

u 

+C18/ (C18+C19) 

R/(E+R) 

C21 : 

(PI) 

u 

+C19/ (C18+C19) 

C,  G,  Deltas  &  Scores 

for  Offeror  A 

Offeror  (C)  Input 

B26 : 

(,D 

u 

0 

Gov't  Est  (G)  Input 

C26 : 

(,D 

u 

0 

Delta 

D26 : 

(PI) 

@IF(C26=0,@NA,(C26-$P$26)/$P$26) 

Delta' 

E26 : 

(PI) 

@IF(C26=0,@NA,(B26-C26)/C26) 

Gov't  Est  Score 

F26 : 

(FI) 

@IF(D26<$D$7,10,@IF(D26<$C$7, 

( (10-$E$7)*($C$7-D26)/($C$7-$D$7))+$E$7,$E$7)) 

Cost  Realism  Score  G26:  (FI)  @IF( @ABS (E26 ) <$D$8 , 10 , @IF( @ABS 

(E26)<$C$8,((10-$E$8)*($C$8-@ABS(E26))/($C$8-$D$8))+$E$8.$E$8)) 


Weighted  Scores  for  Offeror  A 


Gov't  Est 

B35 : 

(FI) 

+F26*$C$1 2/10 

Cost  Realism 

C35 : 

(FI) 

+G26*$C$ 1 3/ 10 

Total 

D35 : 

(FI) 

+B35+C35 

Percent  of  Max 

E35 : 

(P0) 

+D35/(C$12+C$13) 

Tech  Adv  Tradeoff- 

-High 

F35 : 

(P0) 

(R$26-D35)/(C$1 1-R$26+D35) 

Tech  Adv  Tradeoff- 

-Low 

G35 : 

(P0) 

@IF(R35>F35,R35,F35) 

F-46 


Table  9  (continued) 


Delta  Range  (+2%)  for  Offeror  A 


Gov’t  Est  Low 
Gov't  Est  High 
Cost  Realism  Low 
Cost  Realism  High 


B45 :  (PI)  @IF((D26-0.02)<0,0tD26-0.02) 
C45:  (PI)  +D26-rO  .02 

D45 :  (PI)  @IF(E26<0,E26+0.02,E26-0.02) 
E45 :  (PI)  @IF(E26<0,E26-0.02, E26+0 .02 ) 


Score  Range  for  Offeror  A 


Gov’t  Est  Low  B55 :  (FI)  @IF(C45<$D$7 , 10 , @IF(C45<$C$7 , 

((10-$E$7)*($C$7-C45)/($C$7-$D$7) )+$E$7 , $E$7 ) ) 

Gov  t  Est  High  C55:  (FI)  @IF(B45<$D$7 , 10 , @IF(B45<$C$7 , 

((10-$E$7)*($C$7-B45)/($C$7-$D$7))+$E$7,$E$7)) 

Cost  Realism  Low  D55:  (FI)  @IF(@ABS(E45)<$D$8, 10,@IF(@ABS 

(E45)<$C$8,((10-$E$8)*($C$8-@ABS(E45))/($C$8~$D$8))+$E$8,$E$8)) 
Cost  Realist  High  E55:  (FI)  @IF(@ABS( D45 ) <$D$8 , 10 , @IF(@ IBS 

(D45)<$C$8,((10-$E$8)*($C$8-@ABS(D45))/($C$8-$D$8))+$E$8,$E$8)) 

Weighted  Low  F55:  (FI)  +B55*$C$12/10+D55*$C$13/10 

Weighted  High  G55:  (FI)  +C55*$C$12/10+E55*$C$13/10 


Scoring  Plotting  Points 


Gov't  Est  Score  B192:  (FI)  @IF( A192<$D$7 , 10 , @IF( A192<$C$7 , 

((10-$E$7)*($C$7-A192)/($C$7-$D$7))+$E$7,$E$7)) 

Plotting  Point  for  A  092:  (FI)  @IF(  $D$26>*$A192  ,  $F$26 , 0) 

Cost  Realism  Score  B299:  (FI)  @IF(@ABS( A299) <$D$8 , 10 , @IF( @ABS 

(A299)<$C$8,(( 10-$E$8)*( $C$8-@ABS (A299))/($C$8-$D$8))+$E$8,$E$8)) 
Plotting  Point  for  A  C299:  (FI)  @IF($E$26<-$A299#AND#$E$26<0, 

$G$26 ,0) 


Table  10 

TRDTRAP  Model  Listin 


Spread  and  Minimum  Score  Inputs 


1st  Gov't  Est  Base 

B9 : 

(PI)  u 

0 

2nd  Gov't  Est  Base 

C9 : 

(PI)  u 

0 

3rd  Gov't  Est  Base 

D9 : 

(PD  u 

0 

1st  Gov't  Est  Top 

E9 : 

(PI)  u 

0 

2nd  Gov't  Est  Top 

F9 : 

(PD  u 

0 

3rd  Gov't  Est  Top 

G9 : 

(PD  u 

0 

Gov't  Est  Min  Score 

H9: 

(F0)  U 

1 

1st  Cost  Realism  Base 

BIO: 

(PD 

U  0 

2nd  Cost  Realism  Base 

CIO: 

(PI) 

U  0 

3rd  Cost  Realism  Base 

DIO: 

(PD 

U  0 

1st  Cost  Realism  Top 

E10: 

(PD 

U  0 

2nd  Cost  Realism  Top 

F10 : 

(PD 

U  0 

3rd  Cost  Realism  Top 

G10 : 

(PD 

U  0 

Cost  Realism  Min  Score 

H10 : 

(F0) 

U  1 

Weight  Inputs  for  Weight 

_A 

Technical 

C13 : 

(FI) 

U  (C17-C16)/( 1+C20+C2 1 ) 

Gov't  Est 

C14 : 

V  J 

U  J-Cl 3*C20 

Cost  Realism 

C15 : 

(FI) 

U  +C1 3*C2 1 

Other 

C16 : 

(FI) 

U  0 

Total 

Cl  7: 

(FI) 

100 

Weight  Ratios  for  Weight 

_A 

E/T 

C20 

(PD 

U  +C14/C13 

R/T 

C21 

(PI) 

U  +C15/C13 

E( E+R ) 

C22 

(PD 

+C20/ (C20+C21 ) 

R(E+R) 

C23 

:  (PD 

J-C2 1  /  (  C20J-C21 ) 

Approximate  Cost  Input 

C25 : 

:  (CO) 

U  0 

Indifference  Envelope  Inputs 

Gov't  Est  Min  % 

C31 

(PD 

U  +C35/$C$25 

Gov't  Est  Max  % 

D31 

(PD 

U  +D35/$C$25 

Cost  Realism  Min  % 

E31 

(PD 

U  +E35/ $C$  25 

Cost  Realism  Max  % 

F31 

(PD 

U  +F35/ $C$  25 

Gov't  Est  Min  $ 

C35 

(Cl) 

U  +$C$25*C31 

Gov’t  Est  Max  $ 

D35 

(Cl) 

U  +$C$25*D31 

Cost  Realism  Min  $ 

E35 

(Cl) 

U  +$C$25*E31 

Cost  Realism  Max  $ 

F35 

(Cl) 

U  +$C$25*F31 

Table  10  (continued) 


Tradeoff  Plotting  Points  for  Weight  A  and  First  Spread  Follow: 


Gov't  Est  for  Each  Technical  Advantage 

Unadj  Cost  Disadv  C85:  (PI)  ( ( $B$9-$E$9) / ( 1-0 . 1*$H$9) )*$A85/ 

(C$20*(i+$A85))+$E$9 

Adj  Cost  Disadv  F85:  (PI)  @IF( $A85/ ( C$20* ( 1+$A85)  )<(  1-0 . 1 

*$H$9),+C85,@NA) 


Cost  Realism  for  Each  Technical  Advantage 

Unadj  Cost  Realism  C126:  (PI)  ( ( $B$ 10-$E$ 10) / ( 1-0 . 1*$H$ 10 ) ) 

*$A126/(C$21*(1+$A126))+$E$10 

Adj  Cost  Realism  F126:  (PI)  @IF($A126/(C$21*( 1+$A126) )<( 1- 

0.1*$H$10) ,+C126,@NA) 


Gov't  Est/Cost  Realism  for  Each  Gov ' t  Est  Cost  Advantage 

Unadj  Cost  Realism  C167:  (PI)  +$B$10-($B$10-$E$10)*(1-(C$20*(1- 

0.1*$H$9)/(C$21*(1-0.1*$H$10)))*(1-($B$9-$B167)/($B$9-$E$9))) 

Adj  Cost  Realism  F167:  (PI)  @IF(C167<=$B$ 10 , C167 , @NA ) 


Gov't  Est  Band  for  Each  Technical  Advantage 

Unadj  Intermed  196:  (PI)  ( ( $A96/ ( $D$20* ( 1+$A96) ) )*( $C$9 

-$F$9) /( 1-0. 1*$H$9))/ 1.05 

Unadj  Lower  J96:  (PI)  +$C$9-( ( 1-0. l*$H$9-0 . 7*$A96/ 

($D$20))*($C$9-$F$9)/(1-0.1*$H$9)) 

Intermed  K96:  (Pi)  @IF($A96/(D$20*(1-$A96))<(1- 

0.1*$H$9),+I96,@NA) 

Lower  Band  L96:  (PI)  @IF($A96/(D$20*(1-$A96))<(1- 

0.1*$H$9),+J96,@NA) 


Cost  Realism  Band  for  Each  Technical  Advantage 

Unadj  Delta'  1137:  (PI)  ( ( $A 1 37/ ( $D$21*( 1  +  $A137 ) )  )  * 

($C$10-$F$10)/(1-0.1*$H$10))/1.05 

Unadj  Tech  Adv  J137:  (PI)  +$C$10-( ( 1-0. l*$H$10-0. 7*$A137/ 

($D$21))*($C$10-$F$10)/(1-0.1*$H$10)) 

Adj  Delta'  K137:  (PI)  @IF($A137/($D$21*(1+$A137))<(1- 

0.1*$H$10) ,+I137,@NA) 

Adj  Tech  Adv  L137:  (PI)  @IF($A137/($D$21*(1+$A137))<(1- 

0.1*$H$10),+J137,@NA) 
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